How Kensu’s Integration with Matillion empowers data teams to deliver reliable data

It’s a common thread amongst data-driven organizations: data teams face soaring volumes of data with varying complexities, which raise issues regarding data reliability. Efficiently monitoring data pipelines has become paramount to swiftly identifying and addressing potential data incidents, ensuring minimal impact on data practitioners and end users. 

In another common thread, Matillion has become a leader in data productivity. Using their platform, data teams can deliver business-ready data faster by deploying data integration initiatives more efficiently. With its latest Data Productivity Cloud development, Matillion bridges the skills gap of interacting with data pipelines thanks to its Code-optional platform. But as Matillion users work to drive more value by building new pipelines, they must adopt a proactive approach to safeguard the integrity and reliability of the data they deliver.

Enter Kensu

Kensu now supports Matillion users with its data observability solution, which is the first of its kind that integrates with Matillion and seamlessly brings advanced observability capabilities, empowering organizations to gain richer insights about their data pipelines and ultimately strengthening trust in their data. 

Through its integration with Matillion, Kensu retrieves valuable context regarding job executions as well as information about the lineage between data sources used in the execution. In parallel, Kensu connects to database tables to retrieve information about the data (e.g., schemas and metrics) and validate that they pass predetermined rules, feeding this information back to data teams. This way, data practitioners have all the contextual information necessary to understand where data issues are coming from and what is the fastest way to solve them. 

In addition to these data observability capacities that allow data teams to tackle data incidents faster, Kensu also offers a “circuit breaker,” which can halt a data pipeline if a predetermined rule has been broken. This feature is critical for safeguarding the overall health and reliability of the delivered data. 

The Kensu Circuit Breaker: how does it work with Matillion?

Let’s say your organization executes countless jobs across different Matillion projects. Once these jobs are completed, a final job consolidates the data from all of the proceeding jobs. While this data convergence logically tracks, it allows for the risk of your final job running with incorrect data, for instance, any preceding jobs contain data issues. 

To eliminate this risk during the data consolidation process, your team can leverage the Kensu Circuit Breaker in conjunction with the rules in Kensu. This powerful combination will stop the final job’s execution if any of your previous jobs have not been executed as planned, acting as a proactive protection against delivering inaccurate or incomplete data. 

Through the implementation of Kensu's advanced observability features and its Circuit Breaker, we’ve successfully resolved in the example above the reliability issue. Kensu’s integration with Matillion creates confidence in the integrity of the final data output, empowering your organization to make more confident, informed decisions based on up-to-date and reliable information.

Kensu and Matillion in action

Kensu’s integration with Matillion is an important feature for a large American pharmaceutical and health sciences company where data incidents were propagating downstream in its rapidly scaling data environment and impacting end users.

The Kensu Circuit Breaker gives them confidence that they can stop any faulty pipelines, ensuring integrity in the data presented to end users. In parallel, the data team can receive from Kensu all the contextual insights it needs to identify the source of the incidents and troubleshoot them faster.

As a consequence, with Kensu acting as a safeguard against broken data, the organization can now scale with confidence against the threat of broken data and reinforce the trust its end users have in data.

If you would like to experience the Kensu integration with Matillion yourself, book a demo with our team to discover how you can get immediate control over your Matillion jobs’ data.