Semantic is the study of meaning or truth and how they are defined. In data, the semantic layer is the framework where data teams define and validate their business truth in a centralised, structured, manner. This layer is not just about defining metrics; it’s about establishing a single source of truth that ensures consistency and reliability across the entire organisation.
You might wonder, aren’t data analysts and engineers already doing that? Well … yes, but the traditional approach often leads to inconsistencies and requires a lot of resources to maintain. The semantic layer addresses these challenges head-on by providing a unified, accessible system for defining business logic. This not only streamlines data governance but also enhances data scalability and adaptability.
Let’s delve deeper into the story of the semantic layer, explore why it is a transformative tool, and discuss strategies for its implementation.
Understanding the genesis of the semantic layer is enough to realise how important it is for businesses today. The story begins in 2014 with a young data scientist at Airbnb called Nick Handel who found himself overwhelmed by the demands of his product manager. Each new feature introduced required tracking of its performance metrics, a demand that quickly became unsustainable.
Thankfully for Nick, the data engineering team at Airbnb introduced an experimental tool called ERF (experiment reporting framework). This tool revolutionised the way metrics were handled by centralising their definitions, thereby enabling faster experimentation and analysis for product teams. The metric store enabled Nick to track hundreds of features and experiments. He would simply select and experiment with a few lines of code.
Fast forward to 2021, Nick Handel and his friends James, the original PM of the metrics repository in 2014 at Airbnb, and Paul, who built the infrastructure behind it founded Transform. They built an Open Source technology called MetricsFlow which has democratised the metrics layer ever since.
When you don’t spend debating whose truth is right, you spend time executing. The semantic principle is a vital exercise for businesses because once everyone has agreed on what the truth is, collaboration and trust can fully flourish.
With the semantic layer, all business logic is defined and validated in one central place, reducing discrepancies, making it easier and quicker to adjust definitions, and improving trust across the organiwation. When querying a metric like “Churned Users”, everyone has the same definition and the same number, leading to consistent answers across different reports. With APIs, the semantic layer can be used outside of traditional BI tools, freeing up time for other teams, like your data scientists, who work with business metrics but do not consume them through dashboards.
The use of everyday language to define metrics makes them much more accessible and easier to engage with than complex queries. This approachability encourages wider adoption of self-service data tools across the organisation, as team members can more easily interpret and utilise data without needing deep technical expertise.
Implementing a semantic layer streamlines data governance, scalability, and adaptability processes. By reducing the need for repeated updates across numerous dashboards and queries and minimising discrepancies in data interpretation, organisations can achieve significant cost savings in their data operations.
Finally, when you ask “how many customers did we have last month?” and everyone has the same number; that’s magic. Now, how do we implement this? Glad you asked!
The Semantic Layer is built on three basic concepts for creating a metric:
Entities: Think of entities as the core subjects of your data analysis, such as customers, products, or transactions. These are the primary objects around which your data revolves.
Dimensions: Dimensions provide context to your entities. For a customer entity, dimensions could include attributes like age, location, or customer type. These descriptors allow you to segment and analyse your data more effectively.
Measures: Measures are the quantitative elements you’re evaluating, such as average order value, total sales, or customer lifetime value. These metrics offer insights into performance and behaviour.
In early 2023, dbt Labs acquired Transform, Nick’s company. This move brings the semantic layer and metrics store capabilities directly into the dbt cloud ecosystem, offering a streamlined path for organisations to enhance their data analytics practices.
Implementing the semantic layer involves defining these elements within your dbt environment, ensuring that every metric is consistent, accurate, and aligned with your business logic.
The rise of metrics-first BI tools, such as Steep, Tableau, and Preset, enables teams to import metrics directly from dbt cloud, facilitating the integration with the semantic layer. By leveraging such tools, organisations can further democratise data access, allowing non-technical users to engage with data insights through a more intuitive interface.
Our experience with implementing Steep for our clients has been more than positive. The ability to build a metrics catalog directly from your data warehouse or database, without the need for complex SQL queries, empowers teams to focus on strategic initiatives rather than data preparation.
The semantic layer is an exciting turning point not only for data teams but for every stakeholder. By centralising the definition and management of metrics, not only streamlines the work of data teams but also empowers every stakeholder engaged with data across the organisation. This transformative approach enhances trust in data, accelerates the pace of change, and allows teams to focus on leveraging their expertise rather than deciphering complex metrics. More importantly, it is an exercise that pushes teams to truly collaborate.
As data experts, we quickly recognised the value of the semantic layer and how it can radically improve trust and productivity across organisations. If you are interested in hearing more about how the semantic layer can be implemented in your team, do not hesitate to reach out to us directly or sign up for our 3-hour free data assessment.
