Prevent data issues from cascading and deliver reliable insights with Kensu + Azure Data Factory

38% of data teams spend between 20% and 40% of their time fixing data pipelines¹. Combating these data failures is a costly and stressful activity for those looking to deliver reliable data to end users. Organizations using Azure Data Factory can now benefit from the integration with Kensu to expedite this process. Their data teams can now observe data within their Azure Data Factory pipelines and receive valuable insights into data lineage, schema changes, and performance metrics. Ultimately, data teams are equipped to handle data issues, stop them from propagating, and provide reliable data to end users. 

Enter Kensu

Through the integration with Kensu, Azure Data Factory activities are made data observable, and data teams can obtain a comprehensive view of their data pipelines to identify data incidents. The integration functions by retrieving pipeline run information directly from Azure Data Factory, extracting lineage and contextual information, such as pipeline name, activity name, and timestamp. From there, viewers may use these contextualized insights to discern which pipelines produce specific outputs and consume particular inputs.

To enrich these observations, Kensu also retrieves schema details and computes statistical metrics on the data sources used in the activities. This wealth of information provides the full story of the data movement—by acquiring a deeper understanding of data usage and incidents, data teams may more efficiently complete root cause and impact analysis when incidents arise.

The Kensu Circuit Breaker: How does it work with Azure Data Factory?

Let’s say an application processes data before sending it to several pipelines. If the initial application encounters an incident, the flawed data will cascade across all the downstream pipelines, raising the risk for a single incident to spread and adversely impact end users.

Such a risk exists for most data teams, given that only 7% could identify data issues before they impact users². To eliminate potential for data problems, data teams can leverage the Rules in Kensu in conjunction with the Circuit Breaker. In the example above, this powerful combination stops the initial job’s execution if data reliability standards are not met, acting as a proactive protection against propagating inaccurate or incomplete data downstream. This renders data teams the opportunity to resolve the problem by leveraging the insights provided by the Kensu platform.

The Kensu Circuit Breaker is a component that data teams can seamlessly import into an Azure Data Factory environment and incorporate into any desired pipeline. Once in place, the Kensu interface allows activating or deactivating the rules that govern the circuit breaker's operation in a few clicks.

Kensu and Azure Data Factory in action

Azure Data Factory and Kensu users can supply data with the confidence that its integrity is intact. With this integration in place, they can effortlessly fetch data observations for all their Azure Data Factory pipelines and prevent data incidents from cascading, eliminating frustrations that arise when faced with broken data. 

The integration with Azure Data Factory integration will be generally available in November, 2023. However, you can already discover its capabilities with our team by booking a demo on our website or by downloading our two-pager.

Sources: The State of Data Observability 2023.