Logo Zenital
AboutServicesProjectsBlogTraining
Contact us
Community
Contact us
Burger menu icon

Data cleaning and validation before a migration

The challenge
What we did
What we achieved
Cross icon

Data cleaning and validation before a migration

The challengeWhat we didWhat we achieved

A custom ETL process to clean and validate data prior to an important system migration

‍

The challenge

A service-oriented company was preparing for a migration of their data. And, as a previous and critical step, they needed to clean and validate inventory data that came from different sources.

They brought us in to build an ETL process. This began as a technical task, but quickly evolved into a broader collaboration around data architecture and strategic recommendations.

‍

  • Industry: Services
  • Solution type: Data cleaning and validation
  • Tech stack: Microsoft Fabric, Power BI

What we did

The client was dealing with data that came from several sources, with different naming conventions and formats. Our job was to unify and prepare this data for future use.

Our solution: we built an ETL, which follows this process:

  • Multi-source ingestion: We ingest data from files in Excel and CSV, coming from different sources to Microsoft Fabric.  
  • Rule-based cleaning: We developed logic and coded rules for this cleaning stage.  
  • Business validation: The results, with the clean data, are stored on Excel tables and visualized on PowerBI dashboards. The user receives a table with all the data for validation, according to business rules, and we constantly iterate through the process.

‍

What we achieved

Our team succeeded in building a useful ETL for preparing, cleaning, and validating the data prior to the next step: migration.

We implemented the process and, although it was a challenge to work with Microsoft Fabric (since it is a relatively recent product and some of its functions were still in the early steps), it was a successful project: we established the Data Cleansing flow with great results and positive feedback from the client.

This project is a strong example of how data preparation and cleaning ensure long-term success in a strong data architecture.  

‍

“The migration of our system would not have been possible without the assistance of the zenital team. Thanks to the ETL they built to clean and validate our data, and to their assistance in applying business rules, we now have better data quality and a clear and structured process, optimizing our work”.

‍

Key takeaways

  • What business challenge did this project solve?

Our custom ETL, built applying business rules, allowed users to have actionable information about the data so they can make informed decisions to set up a clean foundation before the migration.

  • How was the solution implemented?

We built the pipelines in Microsoft Fabric, ingesting data from different files, coded rules and logic to clean the data, and stored the results, setting up visualizations for the client.

  • What results or impact did the project deliver?

From data coming from diverse sources, with different naming conventions and formats, and with duplicated fields, we delivered a clean, reliable data set, with custom visualizations.

© 2023 zenital. All rights reserved.
Linkedin Icon
Logo Zenital
Data engineering studio since 2016
Pau Claris, 100, 6th floor · 08009 Barcelona

Company

AboutServicesProjectsResourcesCommunityContact us

Legal

Legal notePrivacy PolicyCookies PolicySocial Network Policy