Abbreviations:
- Xplain Data CausalDiscoverer : XCD
- Xplain Data ObjectAnalytics : XOA
- Xplain Data ObjectExplorer : XOE
- Xplain Data AdminTool : XAT
- Causal Discovery (CD)
Content: What is ObjectAnalytics, what it is not and why it is needed for Causal Discovery!
What are best practices / guidelines for a successful data import / analysis?
What is XOA?
- XOA means restructuring data with a certain focus entity in mind (The Patient, the Machine Part, the Employee…). Data entities are treated as objects, instead of flat tables. Check the explanations in the Video or this text. While this restructuring needs a little bit of preparation in the beginning, it will pay off hugely in the long run by having all relevant information for each object readily available, leading to a significant improvement in runtime. It is only with this boost that Causal Discovery can actually be applied on complex real-world data!
Why should I use XOA instead of any standard approach?
- Most real-world data is quite complex and contains relationships between various entities (Diagnoses-Prescriptions, Measurements-Product Quality etc.). While XOA admits a complete 360°-view of the object in focus, flattening this data into a dataframe will always lead to a loss of information or introduce bias!
- Storing data as objects instead of flat tables means that all relevant information for an object is readily available and does not have to be looked up first. With the 360° view it is very easy and natural to build variables or features over multiple (sub-)objects. For instance,
- the sum of Total Prescription Costs after a certain Diagnosis
- the maximum force applied to a Machine part before a certain Production Step.
- the number of Events which have happened for any Item on a Purchase Document
There are virtually infinitely many ways to aggregate data over objects and build features. With all data for one object stored jointly, these can be computed much faster than in a relational database.
- Storing data as objects instead of flat tables / dataframes leads to a significant improvement in runtime and scalability, which is especially important for Causal Discovery. Typical CD algorithms do not perform well on complex real-world data, as there are millions and millions of possible variables/features to be built. Standard algorithms simply cannot handle tables with millions of columns. In fact, the number of variables/columns typically is reduced beforehand by a domain-expert, which is prone to introducing bias! On the other hand, XCD is able to incorporate the whole dataset at hand and hence assuring that no confounders are left out and only then can you be reaonably sure to have found an actually causal relationship. Therefore, ObjectAnalytics makes Causal Inference feasible in a complex real-world environment.
How does my data have to look like such that XOA is as performant as possible?
- XOA particularly thrives when using complex real-world data, for example from a relational database. The more sub-objects are appended to a root-object, the more XOA can show its strengths by computing meaningful features over these sub-objects. Typically, these sub-objects are streams of events (diagnoses, prescriptions, measurements etc.). The best-case scenario is that each of these sub-objects also has timestamps appended to it. This ensures that events will be considered in the correct (causal) order!
How does XOA represent my data? What is an Object Tree?
- Each table in the source database can be regarded as an object. One of these has to be defined as the root object and others may be appended as sub-objects. For example, the “Patient”-object is suited as the root, and it has the “Diagnoses” and the “Prescriptions”-object appended to it as sub-objects. Each of these objects is characterized by its own dimensions and attributes.
Such a hierarchical representation of the data is called an ObjectTree. Columns in the table are added to the corresponding objects and are called dimensions. These dimensions can be represented / discretized by various attributes (see also 2. An introduction to the XOE — Xplain Data Manual documentation).
Couldn’t I also represent the same data as a flat table?
- Somehow you could, but you would probably already be forced to transform the data in some way. In particular, there is no unique way to do it, so you actually run the risk of introducing a bias into the data. Each object itself can be represented easily as a table, but how do we append the sub-object data to the root object? Do we have to append a column for each single diagnosis and each timestamp, say
and also do the same for all prescriptions? Note that every patient has a different number of diagnoses and prescriptions, so we would have to consider the patient with the most diagnoses/prescriptions and create that many columns. This means, firstly, that the table will have a huge number of columns and, secondly, that many columns will stay empty for many patients / rows.
- Another way to do it would be to already compute aggregations/features from the data, for example, the number of diagnoses, the sum of total prescription costs etc.
- In other words, while the object-centric view allows a whole 360° view of the object in focus, a flat table representation of the same data will always be like a 2-dimensional projection of a 3-dimensional object! Flattening data introduces bias!
Where can I define my ObjectTree?
- By creating an ObjectView in the AdminTool (XAT) (see the docu material there or here). Simply toggle the Developer Mode and start it from the XOE.
How do I have to preprocess my data before being able to create an ObjectView?
- If there are no inconsistencies (for example inconsistent data types within a column) in the data, the only thing to check is if there are unique IDs, which can connect an object to its parent(s). If this is not the case, these IDs might have to be created manually. Typically, this can be done by joining one or more column names to a new one. Otherwise, no preprocessing has to be done. The data can be loaded into the XOE as a whole, we don’t even have to worry about missing values!
What do I have to think about before defining an ObjectView?
- The input needed by the user is the hierarchical structure of the data. This means in particular, which of my available objects is the main object in focus, and in which order do I want to append the sub-objects. Typically, a domain expert will have a good intuition for a meaningful data structure, but it also usually just follows from any particular analysis question you have in mind - Why do Patients get sick? Why do I have Produced Parts that are broken? Why do Customers churn? Why is my Process non-compliant etc.
Can I redefine my root-object on the fly? How flexible is the ObjectTree?
- Object Trees cannot be redefined directly in the XOE. They have to be defined first with the AdminTool via a so called ObjectView. Another ObjectView has to be created to that end. However, via the AdminTool, an already existing ObjectView can easily be redefined to alter its hierarchy (by redefining the keys).