What are the challenges of using a data vault?
From flexibility to scalability and efficiency, using Data Vault as your data modelling approach has many benefits. But simultaneously there are challenges that you need to be aware of. In this blog I’m going to walk you through the limitations and how you can overcome them.
The approach Data Vault takes when modeling data (something I will go into detail on further down) results in a significantly larger amount of data objects compared to other approaches. These objects include things like tables and columns and the reason there are so many more is because Data Vault separates information types.
As a consequence, the up-front modeling effort can be larger to accommodate the resulting benefits – mentioned above – as the end result. It also means that during the modeling process there can be larger numbers of manual or mechanical tasks involved to establish the flexible and detailed data model with all its components.
How can data vault limitations be addressed?
To avoid time-consuming manual tasks during the modeling process, architects can automate parts of the model, making it more efficient to create, update and maintain long-term.
How can they do that?
Within the Data Vault approach, there are certain layers of data. These range from the Source Systems where data originates, to a Staging Area where data arrives from the source system, modeled according to the original structure, to the Core Data Warehouse, which contains the raw vault, a layer that allows tracing back to the original source system data, and the business vault, a semantic layer where business rules are implemented. Finally, there are data marts, which are structured based on the requirements of the business. For example, there could be a Finance data mart or a Marketing data mart, holding the relevant data for analysis purposes.
Out of these layers, the staging area and the raw vault are best suited to automation.
What are the characteristics of Loading...data vault modeling?
The data vault modeling technique brings ultimate flexibility by separating the business keys, which uniquely identify each business entity and do not change often, from their attributes. This results, as mentioned earlier, in many more data objects being in the model, but also provides a data model that can be highly responsive to changes, such as the integration of new data sources and business rules.
The basic structure of the model comes from the business keys and the relationships between them. Their stable nature provides the key ingredient for a robust data model, but also means the keys need to be chosen carefully, as they form the very basis from which everything else is derived.
Hubs
The tables which contain the business keys are called hubs in the data vault approach. In addition to storing the keys, hubs also contain surrogate keys and metadata for each business key. Finally, the source of each business key can also be found in the hub, so that information can be traced back to its origins.
Links
Link tables are many-to-many join tables that connect different business keys. Within link tables the information you will find are the surrogate keys for the hubs connected via the link, as well as the surrogate key for the link and the metadata about where the association originated from.
Satellites
With the hubs and links in place, the structure of the data vault model is set up. It does not, however, contain any attributes yet. This is where satellites come in. Satellite tables hold metadata that connect them to their parent hubs and link tables. They also contain metadata about the origins of the attributes, as well as temporal attributes. This means that thanks to satellites, data architects can ensure that history is recorded at any interval, while also providing an audit trail and traceability to the source system.
How does Data Vault work with Exasol?
In Exasol you have a database that enables you to work flexibly with a plethora of tools and methodologies so you can choose the right approach for your business and overarching analytics strategy.
Exasol fully supports you in choosing the data modeling technique that best fits your strategy. This means, you can easily benefit from the advantages Data Vault brings.
We have partners such as Datavault Builder and Wherescape who have created data modeling and warehouse automation tools that integrate effortlessly with the Exasol database.
You can also build your data model directly in our database, using our UDF Framework.
Bringing Exasol’s performance to your Data Vault modeled data
Modeling your data in Data Vault can result in complex SQL queries being executed in your data warehouse. Exasol’s architecture and pure design ensures that the outstanding performance we promise to you is sustained throughout the entire data lifecycle, and that includes your data modeling and warehousing processes.
You can audit and reproduce historical query results quickly and efficiently, while also loading all your large data volumes into the warehouse and invite your analysts and data scientists to run their workflows, analyses and analytical models directly in the data warehouse without sacrificing speed and reliability.
Our partnerships with Datavault Builder and Wherescape focus on improving the user experience with every new release, which is why we take your feedback and jointly work on the continuous development and integration of our respective products.
Internally, we have been using Datavault Builder for our own data warehouse development and you can watch this video to get an impression of our partnership with the team at Datavault Builder.
Where to from here?
If you want to start right away, you have come to the right place. The easiest way to test things out is to use our trial version of the Exasol database and load some of your own data into it. If you already have a data model in place and want to test how it would perform in Exasol, you can do that as well.
To go through these steps, here are the relevant resources:
- Download and install our free trial or get setup with a trial in our ExaCloud.
- Use our documentation portal if you need help getting connected to your existing tools, e.g. Datavault Builder.
- Join our online Community where you can get help and tips, learn new tricks and connect with like-minded people. Plus, you can get in touch with our people and let us know how you’re doing during your trial.
We look forward to working with you and to helping you succeed with Data Vault modeling in Exasol.
Eva Murray, Technology Evangelist, Exasol