What Is Data Mesh?

In an age where most companies purport to take a “data first” approach, not all are using that data to the best of its scalability. You may have heard the term “data mesh” thrown around as a hot topic among analysts. What is data mesh? It’s a new approach to data management that aims to make data more accessible and scalable across users. 

Here’s how it works. 

What is data mesh?

Data mesh is an enterprise data architecture first created by Zhamak Dehghani in 2019. It refers to a decentralized data architecture that organizes business data by specific domains. For example, data can be separated into different domains for marketing, HR, customer service, and other business units. Each team has ownership of its own data.

Data mesh represents a new paradigm of data management, where data can be thought of as a product to fulfill key tasks within the business. For example, data might improve decision-making, detect security risks, or alert businesses to key changes in their environment.

Dehghani compares this approach to more traditional, monolithic data architecture, where data is centralized or stored as a “data lake.” A typical failure of this data architecture is that it doesn’t support enterprises effectively, failing to account for their large number of data sources and wide range of consumers. As more data becomes available, the ability to harmonize it in one place effectively diminishes. Data integration and governance have become more difficult.

Data mesh uses a hierarchical data warehouse to store data in folders or files, whereas data lake has a flat data storage architecture.

What are the advantages of data mesh?

To be clear, data mesh is not a form of data silo. In fact, the “mesh” idea is to make data more available and to directly connect data owners, consumers, and producers. With this link between data producers and business users, IT can increasingly be removed as a “middleman” from projects that require data resources.

Compare that to monolithic data architecture, and you’ll find IT almost always acts as a go-between for disseminating data resources. In this sense, data mesh provides efficiency and accessibility for all parties.

Improve speed to innovation

Another advantage of data mesh is that companies can speed up their innovation cycles. They can do this by shifting from the manual, batch-oriented ETL to a more efficient CTL approach. This represents a huge reduction in data engineering and improved support for no-code and self-service pipelines.

Boost agility

Data mesh can help boost agility, with each node working independently. Data owners and consumers can focus more on their individual use cases rather than being caught in the weeds of technical complexity.

Data interoperability

Interoperability and standardized communications are other advantages of data mesh. Each domain maintains a universal set of data standards that facilitate communication between them when necessary. There are plenty of examples of data being valuable for more than one domain, so cross-collaboration, where data mesh architecture standardizes core aspects of data features, is critical.

Is data mesh the right architecture for your business?

The basic premise of data mesh is that individual business domains should control, access, and define their own data products. Stakeholders within those domains tend to understand their data needs better than anyone else. If they must work with data engineers outside of their domain, as often happens in a data lake, it can be time-consuming and ultimately an ineffective way to work.

However, that doesn’t mean data mesh is the right architecture for all organizations. In some cases, where the size of the data ecosystem is relatively small, data mesh may be an overkill approach. 

On the other hand, organizations with a combination of a large quantity of data sources, multiple functional teams (or data domains), frequent data engineering bottlenecks, and a high priority for good data governance may be the biggest beneficiaries of a data mesh approach.

Conclusion

What is data mesh? For businesses that have outgrown traditional monolithic data architecture, data mesh is a solution to cut back some of the typical failings of this approach. Data mesh is a hot topic in data engineering and is set to transform data management in businesses that aim to operate with data first but need help with engineering bottlenecks.

Importantly, effective data management can be the catalyst for a competitive edge in business. Data mesh aims to democratize data and broaden access beyond an organization’s data engineers and other technical resources. Discoverable, scalable data leads to timely decision-making and improved agility.

Semarchy participated in a special round-table webinar with DBTA, exploring data mesh and data fabric architectures. Sign up here to view the replay.