A custom ETL process to clean and validate data prior to an important system migration
The challenge
A service-oriented company was preparing for a migration of their data. And, as a previous and critical step, they needed to clean and validate inventory data that came from different sources.
They brought us in to build an ETL process. This began as a technical task, but quickly evolved into a broader collaboration around data architecture and strategic recommendations.
- Industry: Services
- Solution type: Data cleaning and validation
- Tech stack: Microsoft Fabric, Power BI
What we did
The client was dealing with data that came from several sources, with different naming conventions and formats. Our job was to unify and prepare this data for future use.
Our solution: we built an ETL, which follows this process:
- Multi-source ingestion: We ingest data from files in Excel and CSV, coming from different sources to Microsoft Fabric.
- Rule-based cleaning: We developed logic and coded rules for this cleaning stage.
- Business validation: The results, with the clean data, are stored on Excel tables and visualized on PowerBI dashboards. The user receives a table with all the data for validation, according to business rules, and we constantly iterate through the process.
What we achieved
Our team succeeded in building a useful ETL for preparing, cleaning, and validating the data prior to the next step: migration.
We implemented the process and, although it was a challenge to work with Microsoft Fabric (since it is a relatively recent product and some of its functions were still in the early steps), it was a successful project: we established the Data Cleansing flow with great results and positive feedback from the client.
This project is a strong example of how data preparation and cleaning ensure long-term success in a strong data architecture.
“The migration of our system would not have been possible without the assistance of the zenital team. Thanks to the ETL they built to clean and validate our data, and to their assistance in applying business rules, we now have better data quality and a clear and structured process, optimizing our work”.
Key takeaways
- What business challenge did this project solve?
Our custom ETL, built applying business rules, allowed users to have actionable information about the data so they can make informed decisions to set up a clean foundation before the migration.
- How was the solution implemented?
We built the pipelines in Microsoft Fabric, ingesting data from different files, coded rules and logic to clean the data, and stored the results, setting up visualizations for the client.
- What results or impact did the project deliver?
From data coming from diverse sources, with different naming conventions and formats, and with duplicated fields, we delivered a clean, reliable data set, with custom visualizations.

.png)