Data Quality Engineer

Job Summary :

We are seeking an experienced Data Quality Engineer (Data QE) to ensure the quality, accuracy, reliability, and integrity of data across enterprise data platforms.

The ideal candidate will have strong experience in Data Warehousing, Medallion Architecture, ETL/ELT testing, Data Governance, Data Quality frameworks, Data Lineage, and test automation.

This role will be responsible for validating data pipelines from multiple source systems such as CDP, GA4, BigQuery, and Azure Blob Storage into Snowflake, and ensuring accurate reporting in Power BI.

Required Experience :

- 5-7 years of experience in Data Quality Engineering, Data Warehouse Testing, or Data Validation.

- Strong hands-on experience with PostgreSQL, Microsoft SQL Server, Snowflake and BigQuery.

- Experience validating enterprise Data Warehouse solutions and Medallion Architecture implementations.

- Proven experience performing Source-to-Target validation from CDP, GA4, BigQuery, Blob Storage, PostgreSQL, and MSSQL into Snowflake.

- Experience in validating Power BI reports, dashboards, semantic models, and KPIs.

- Strong understanding of ETL/ELT data processing, data transformations, and data lineage.

- Experience working with Data Governance, Data Quality, Metadata Management, Data Lineage, Data Dictionary, and Data Catalog solutions.

- Experience with Data Quality tools such as Soda or equivalent.

- Experience designing, executing, and automating test cases for large-scale data platforms using SQL, Python, and PySpark.

- Experience integrating automated testing into CI/CD pipelines using Azure DevOps or GitHub Actions.

Key Responsibilities :

Data Validation & Testing :

- Perform end-to-end data validation across the data lifecycle, including ingestion, transformation, storage, and reporting.

- Validate data movement from source systems including CDP, Google Analytics 4 (GA4), Google BigQuery, Azure Blob Storage, PostgreSQL, MSSQL, and Snowflake.

- Perform comprehensive Source-to-Target (S2T) data validation between source systems and Snowflake.

- Validate data accuracy, completeness, consistency, uniqueness, and timeliness across datasets.

- Execute data reconciliation and data profiling activities.

- Conduct source-to-target validation across databases, data lakes, and data warehouses.

- Validate schema changes, constraints, indexes, stored procedures, functions, triggers, and views.

Medallion Architecture Validation :

- Validate data across Bronze, Silver, and Gold layers within the Medallion Architecture.

- Verify data transformations, cleansing, aggregations, and business rules between layers.

- Ensure data quality controls are enforced throughout the Medallion framework.

ETL/ELT Testing :

- Test and validate ETL/ELT pipelines and data transformations.

- Validate business mappings, transformation logic, and data lineage.

- Verify incremental and full-load processes.

- Perform regression, integration, functional, and end-to-end testing of data pipelines.

- Identify and troubleshoot data discrepancies across source and target systems.

Snowflake Data Validation :

- Validate data ingestion, transformations, and storage within Snowflake.

- Perform large-scale data validation using SQL and automation scripts.

- Validate Snowflake views, tables, materialized views, stored procedures, and data sharing mechanisms.

Power BI Testing :

- Validate datasets, data models, measures, KPIs, and dashboards.

- Ensure Power BI reports accurately reflect Snowflake data.

- Verify report-level calculations, filters, row-level security, and business metrics.

- Perform end-to-end testing from source systems through Snowflake into Power BI.

Data Governance & Data Management :

- Validate implementation of Data Governance policies and standards.

- Ensure compliance with enterprise data quality and governance requirements.

- Validate and maintain Data Lineage, Data Dictionary, Metadata Management, Business Glossary, and Data Catalogs.

- Collaborate with Data Governance and Business teams to improve data quality processes.

Data Quality Frameworks :

- Implement and execute data quality checks and monitoring solutions.

- Define and monitor Data Quality KPIs and metrics.

- Develop and automate validation rules for Completeness, Accuracy, Consistency, Uniqueness, Validity, and Timeliness.

- Support root cause analysis and remediation of data quality issues.

Test Automation :

- Develop automated data validation frameworks and reusable test suites.

- Automate source-to-target reconciliation and regression testing.

- Integrate automated data quality tests into CI/CD pipelines.

- Build automated validation using SQL, Python, PySpark, and Data Quality tools.

Required Technical Skills :

Databases & Data Warehousing :

- PostgreSQL, Microsoft SQL Server (MSSQL), Snowflake, BigQuery, Data Warehouse Testing.

Data Modeling Concepts :

- Star Schema & Snowflake Schema, Slowly Changing Dimensions (SCD), Fact & Dimension Validation, CDC Validation, Data Migration Testing.

SQL Expertise :

- Advanced SQL Query Writing, Complex Joins, Window Functions, Aggregations, Query Optimization, Data Reconciliation Queries, Data Profiling and Data Analysis.

Modern Data Platform Validation :

- Medallion Architecture Validation (Bronze, Silver, Gold), Source-to-Target Testing, ETL/ELT Testing, Data Lake & Data Warehouse Validation, Power BI Report & Dashboard Validation.

Skills

Data Quality, Data Warehousing, Medallion Architecture, Data Validation, Snowflake DB, BigQuery, Power BI, ETL, Data Lineage, Data Governance, Python