Logo Zenital
AboutServicesProjectsBlogTraining
Contact us
Community
Contact us
Burger menu icon

Data Quality First: migrating a data validation framework

The challenge
What we did
What we achieved
Cross icon

Data Quality First: migrating a data validation framework

The challengeWhat we didWhat we achieved

A process of safely transitioning a data quality flow for a complex architecture

‍

The challenge

Isdin, a global Spanish pharmaceutical company, needed to centralize data validation for information coming from multiple countries and teams. They were using Soda, but this was resulting in considerable costs. The goal was to find the right tool to centralize quality checks, detect errors fast, and deliver actionable alerts.

‍

  • Industry: Pharmaceuticals
  • Solution type: Data quality
  • Tech stack: Soda, Great Expectations, Google Cloud Platform

‍

What we did

First, we benchmarked the approach and proposed alternatives. Our recommendation was moving their data quality process to Great Expectations, because of its version control capabilities.

Then, our team:

  • Validated the migration with a PoC in record time: We kicked off a proof of concept to estimate and port the required checks. During the process, we implemented the checks within the expected window and scheduled a parallel run where both tools operated side-by-side to avoid risks, with strong results.
  • Embedded the new tool in the client’s tech stack: We integrated Great Expectations with Isdin’s Google Cloud Platform environment.
  • Rerouted alerting for richer contexts: Alerts that were previously viewed in Soda and sent via Slack now land directly in GCP, providing richer metadata and faster automation.
  • Rolled into production: As Great Expectations moved into production, pipelines continue in Soda for a limited time to provide extra safety during the transition.

‍

What we achieved

  • Centralized and consistent data quality framework: This reduces the risk of errors and builds trust in future analytics.
  • Clearer alerts for faster actions: Alerts in Slack carry richer context now, so the team can take deliberate actions based on the type of error.
  • Cost optimization: Moving to Great Expectations reduced total cost of the Soda service.
  • Low-risk migration: The parallel run ensured consistency and a safe transition.

‍

© 2023 zenital. All rights reserved.
Linkedin Icon
Logo Zenital
Data engineering studio since 2016
Pau Claris, 100, 6th floor · 08009 Barcelona

Company

AboutServicesProjectsResourcesCommunityContact us

Legal

Legal notePrivacy PolicyCookies PolicySocial Network Policy