When Do You Need Data Observability?

Almost every organization is a data organization. Yet, as data proliferates, data ecosystems have become increasingly complex, making it harder for organizations to control and prevent data quality issues from occurring.

Data Observability gives organizations the ability to monitor data usage throughout the data ecosystem. It enables organizations to identify, resolve, and prevent data issues much faster and more easily—especially within complex data systems—because it provides visibility into data usage within data sets, applications, and the system as a whole.

While Data Observability is a new concept to many organizations and data teams, it is a best practice that every organization should apply to their data projects. In this article, we’ll look at when and how to introduce Data Observability into any data project not only to ensure, but accelerate your return on investment.

When to implement Data Observability?

There is the ideal in any data engineering environment, and then there is the reality. Ideally, organizations would be wise to implement Data Observability at the start of their digitalization journey. However, this ideal is only achievable for new companies, as almost all existing companies are already somewhere along the path of digitization and creating a data-driven organization.

For most organizations, the best practice would be to implement Data Observability at the next possible moment. Whether your organization is starting a new data project that will involve ingesting, consuming, or transforming data or adding a database, a data warehouse, or a data lake, you would want to include Data Observability as one of the key requirements for the project.

What is the threshold for implementing Data Observability?

If you prefer to wait to implement Data Observability due to other priorities, it is important to understand when you’re at the tipping point moment. By tipping point or threshold, we are referring to the moment when not adding Data Observability will mean you incur significant technical debt.

Once you begin to integrate data continuously, you will need Data Observability as manual processes will no longer be adequate. Indications that you have reached the tipping point for Data Observability include:

• You have a data provider sending data at a high pace every day
• You are ingesting data without human intervention
• You are delivering data to an algorithm automatically

In any of these scenarios, your data processes have reached a point where you no longer have human eyes on what’s happening. As a manner of speaking, your data has been released into the wild. It is running without your oversight, which means it can also fail without your notice. Data Observability ensures that you have the visibility to know where and when a data issue is occurring.

If you miss this tipping point moment and don’t introduce Data Observability, you’ll be stuck using patches to resolve data issues. Because when data issues aren’t anticipated during the development phase and errors happen after deployment, you are left trying to recover visibility after the fact.

As a workaround, you’ll have to resort to time-intensive procedures to troubleshoot, such as adding extra scanners to try and regain some visibility and then reviewing data logs while trying to reverse engineer the entire application to determine where the problem is occurring. Ultimately, you’ll be trying things randomly to see what might work to resolve the issue because you won’t have precise knowledge of where the issue is or what’s causing it. Whatever patch you implement will add to your technical debt because it’ll be nonstandard, requiring extra effort to maintain. This will diminish your team’s overall productivity, making it difficult to scale because you won’t have the resources to support additional data projects.

However, if you implement Data Observability before you reach this tipping point, you’ll not only have the visibility to identify and resolve issues at the root, but you can do so quickly, easily, and preventatively. You can also apply best practices that increase efficiency and productivity because everything is standardized, allowing you to scale.

What components do you need when first implementing Data Observability?

First, your data team needs to conceptualize what Data Observability should look like for the particular data project you are implementing Data Observability. At this stage, no tool is required, but it is important for all project stakeholders—data engineers, analysts, and business users—to communicate and reach an agreement about the primary objectives the organization wants to achieve with the data project and to define the KPIs to measure the success of the project.

This process of conceptualizing the purpose and requirements of the data usage should begin as soon as you decide to purchase, ingest, or produce new data. Whether you’re building a data warehouse for financial reporting, creating audience segments for a marketing PPC campaign, or subscribing to a new data provider and need to integrate the data into your data lake, you need to make sure that you’re receiving the right data to produce the right results as defined by your objectives and KPIs.

For instance, if you’re a large retailer using data to produce financial reports on individual stores’ sales performance, you’ll want to have a daily view of sales at each location. This will require having fresh data every day—meaning your data needs to refresh every 24 hours at a minimum. Additionally, you’ll want to consider what types of data results might raise red flags—such as a 30% or greater difference in day-to-day sales performance.

As part of the conceptualization phase, all stakeholders of the data project will need to agree on what you need to monitor or observe. Based on your data projects objectives and KPIs, this may include the following data characteristics:

• Freshness: Is your data up to date and being ingested at the correct intervals?
• Volume: Are you receiving the correct amount of data?
• Accuracy: Are the values within each field of a database record correct and accurate?
• Consistency: Is the data being recorded in the same manner across all systems and formats?
• Completeness: Does the data contain all the necessary and expected information?
• Uniqueness: Is your data free of duplication?

In addition to defining what you want to observe about the data, you also want to discuss what data outcomes might indicate issues with the data quality. These outcome constraints could be flagging data results with a greater than 30% discrepancy in the data, too many nulls, or results outside a mean average. For instance, if the retailer in the previous example received a financial report where the sales figure for a store was negative or showed greater than a 30% difference in sales data from the previous day, this would likely indicate an issue with the data.

Data Observability can monitor and detect many instances of data quality issues on its own, helping you identify issues that you hadn’t considered in your initial conceptualization of the data project. However, it is not a fully automated process. Consequently, spending the time to create custom rules and indicators of data quality issues at the onset of your data project will accelerate the value of the Data Observability platform because it will identify more quickly when something is off and why.

Building a culture of Data Observability

We understand that you may not take our “ideal” suggestion to introduce Data Observability at the onset of your data project. But we believe that once you begin to implement Data Observability and see the benefits it provides—faster time to resolution, higher reliability, easier maintenance, and easier to scale data projects—you will begin to see the value of introducing Data Observability earlier in the process.

As you discover more benefits and become keener on introducing Data Observability earlier in the process, you will create a culture of Data Observability. It will become a standard requirement of any data project and be intuitively perceived by stakeholders as having enough value to be worth implementing. Once this happens, you’ll not only reap a strong return on your investment in your Data Observability platform but build greater confidence across the organization in the quality of the data.

Interested in learning more about Data Observability? Read our O'Reilly Report about Data Observability.