Databricks Quality Analyst - Data Reconciliation
First seen by Alion on Oct 30, 2026.
We are looking for a Data QA / Reconciliation Engineer with hands-on experience in Databricks, SQL, and ETL testing. The role involves validating data pipelines, reconciling source and target datasets, verifying transformation logic, and ensuring data accuracy and consistency across data processing layers. The ideal candidate should have strong SQL skills, practical Databricks and Delta Lake experience, and the ability to identify and resolve data quality issues.
Responsibilities :
- Perform end-to-end data validation and reconciliation across source systems, staging tables, and target datasets in Databricks.
- Validate ETL/ELT pipelines, data transformations, and source-to-target mappings.
- Write complex SQL queries using joins, subqueries, aggregations, and data comparison techniques to validate record counts, totals, completeness, and accuracy.
- Identify and investigate data mismatches, missing or duplicate records, and transformation errors.
- Execute data quality checks across Databricks Delta tables and Bronze, Silver, and Gold data layers.
- Design and execute test cases for data integration, functional, regression, and end-to-end testing.
- Use Python/PySpark to automate data validation and reconciliation where required.
- Validate schema changes, business rules, derived fields, and incremental data loads.
- Log defects, perform root-cause analysis, and collaborate with data engineering teams to resolve issues.
- Maintain test scripts, reconciliation reports, test evidence, and data quality documentation.
Requirements :
- 4 - 7 years of experience in data testing, ETL testing, data quality assurance, or data reconciliation.
- Strong hands-on SQL skills, including complex joins, aggregations, subqueries, and source-to-target data comparison.
- Practical experience with Databricks and Delta Lake.
- Good understanding of ETL/ELT pipelines, data transformations, data warehousing, and source-to-target mapping.
- Experience designing test cases, executing regression tests, tracking defects, and validating data quality.
- Working knowledge of Python/PySpark for test automation and data validation is preferred.
- Exposure to AWS data services, cloud data platforms, incremental data processing, and automated data quality frameworks is an advantage.
- Familiarity with CI/CD processes, Agile methodologies, and tools such as Jira is preferred.
- Strong analytical, troubleshooting, and problem-solving skills.
Skills
Databricks, SQL, ETL, Delta Lake, Python, PySpark, Data Quality, Data Warehousing, Data Validation, Testing, ETL/Datawarehouse Testing