Practical Ontologies & How to Build Them - Part 3
Part 3: Objects, events and time series in Palantir Foundry
This is the third installment in a four-part series on how to apply ontological thinking to help operationalize your data using Palantir Foundry. In part two we introduced the Foundry ontology and explored some key considerations to bear in mind when modelling an organization’s data. In this installment, we’ll dig deeper into the differences between objects, events and time series, highlighting when to use each in the pursuit of a practical, easy to understand ontology.
Objects vs Events vs Time Series
Ontologically, objects, events and time series are considered distinct from each other with the following general definitions:
Objects persist through time so if any of their properties evolve over time, previous values are overwritten. For example, a “Person” has an increasing “Age”.
Events occur at a point in time (or span a period) and have properties captured at or during that time which subsequently do not change. For example, a “Journey” has a “Start Time”, “End Time” and resulting “Duration”.
Time series provide a history of an object’s property values. For example, a “Sensor” object with a “Temperature” property that changes over time would generate a time series storing the values and corresponding date/times of each change.
Generally speaking, events and time series are temporal and immutable, whereas objects are mutable. This means that once a time series has recorded a value, or once an event has concluded, their associated property values can no longer change whereas an object’s properties evolve in place over time. While this is a sensible way to think about the differences between the components of an ontology, there are edge cases. For example, imagine a Journey event with a property called Status. This hypothetical property could change over time as the Journey takes place, beginning as “in progress” and eventually updating to “complete”. Generally speaking, however, properties that capture data during the event (such as average speed or distance traveled) do not change once the event has concluded.
This diagram visualizes how objects, events and time series evolve over time:
In Palantir Foundry, both objects and events are modeled as “Object Types”. The ontological distinction is instead made in the choice of properties to add to each object type. For example, if we create a “Rail Journeys” object type, we can add properties of Start_Timestamp and End_Timestamp to ensure it can be used as an event in downstream applications. You can also tell Foundry this is an event explicitly by calling out the start and end time properties during the object’s configuration in the Capabilities tab for the object in the Ontology Manager. This is typically optional as some Foundry applications such as Vertex require events to be configured this way, while most do not. In general, if you want to use the event in combination with other temporal information such as time series properties (e.g. overlay your events on a time series chart or use the event boundaries to filter some time series data) it is necessary to explicitly configure them as events in the ontology.
Time Series Properties
Time series are handled in Foundry via the use of “Time Series Properties”, which are properties of objects that store the historical values of that property over time. For example, the “Fuel Level” of a vehicle as it undertakes journeys.
In the development of a Foundry Ontology, there is often a decision to be made regarding whether to capture temporal data by adding a time series property to an object or by creating an entirely new event-like object type linked to the object related to the event.
For example, consider again our “Vehicle” object type:
Imagine we have a dataset containing the journeys each vehicle has taken, with the following properties:
We have three options when modelling a dataset like this in a Foundry ontology:
We create a “Journeys” object type, which is an event, ontologically. Whether or not this is practical will depend on the scale of the backing data. If this table includes a few hundred to a few million Journeys then this is likely a sensible approach. This is often a simple way to model the data that reflects how it is stored in its source system.
We create an aggregated view of the data such as “Daily Vehicle Journey Metrics” (keyed by day and vehicle) which stores aggregations of the journey data for a single day and vehicle. These aggregations might include things like “average fuel consumed per journey” or “total seconds active during the day”. This approach can save compute and storage if the data scale is large and if the downstream use cases can be achieved using only aggregate metrics.
We extract time series properties from the event table and add them as Time Series Properties to the Vehicle Object Type. We would potentially do this for two reasons:
The data scale necessitates it (as time series properties are efficiently stored and queried in Foundry) or,
The downstream use cases would benefit from the data being in a time series format. For example when many derived time series need to be created using the raw time series as input (derivatives, integrals or other such temporal calculations).
If we want to put the data into Time Series Properties on the Vehicle Object Type, we first need to extract time series datasets from the journey data. These datasets store the value of a given property over time. For example, a dataset storing each vehicle’s “fuel level”, extracted from the starting_fuel_level and ending_fuel_level properties of each journey, as shown below.
Note that this is quite a sparse time series. Generally we’d be looking for a data source that provides more regular updates to our fuel level during our journeys (say, every second). For example via a sensor on the vehicle. However, we’ll use this for now to demonstrate how time series properties are created and attached to objects in Foundry.
Foundry requires that each time series has a unique identifier. In this instance that is the Timeseries_ID column which is a combination of the time series name (fuel level) and the primary key of the vehicle the time series is for (“1” or “2” in this example).
To connect this time series to our Vehicles Object Type, we would add a Fuel_Level property to each vehicle, storing the series id specific to that vehicle and their fuel level. This would direct Foundry to attach the relevant rows of the Fuel Level Time Series (shown above) to the individual vehicles contained in our Vehicle Object Type.
Storing the fuel level of each vehicle as a time series (as opposed to starting_fuel_level and ending_fuel_level properties of our journeys) changes the potential use cases for the data in Foundry. Which method is “correct” depends on the requirements of the data and the solutions being targeted.
Generally speaking, time series properties are efficient both from a storage and compute point of view and so should be utilized where they are appropriate for a use case. They also follow a consistent format which makes them easy to work with; no matter the object in question or the property the time series is tracking, in all cases they are simply a set of values taken at points in time. As a result, they can clean up your ontological landscape by removing the need for some of your event-like Object Types, replacing them with time series properties on your objects.
Given the various operational tools available in Foundry and the fact that some work well with time series properties and some work well with event-like objects, a common real-world approach is to model temporal data in both ways simultaneously. E.g.
Create an event-like “Journeys” Object Type, and
Create time series properties on the associated “Vehicles”.
Some of these time series may have been extracted from the Journey data, and others may have come from other source systems such as a database storing sensor readings.
This way, the data will be available for a variety of possible downstream use cases.
In the fourth and final part in this series, we’ll explore the use the verbs in the Foundry Ontology (or “Actions”) to effect change and trigger business processes. These are a crucial consideration in building your ontology and generally unlock a new dimension of value by capturing the result of human decisions back into the ontology itself.
With 100+ years of combined experience, Fourth Age is a leading consultancy focused on delivering value with Palantir Foundry and AIP. We provide a full range of services to ensure our partners can maximize the value of their strategic software investments, including strategic consulting, end-to-end use case delivery and long-term enablement. If you’d like to learn more about how we can help you unlock the potential within your organization’s data, reach out to us via our website.