Sr. Data QA Engineer

Sr. QA Data Engineer

Employment Type: Full-Time, Permanent

Location: Remote – Pan India

Shift Timings: 11:00 AM – 08:00 PM IST

Reporting To: Sr. Technical Manager / Director of Engineering, as assigned by Management

Position Overview

We are looking for an experienced Sr. QA Data Engineer to join our Data Engineering team and take ownership of the quality, accuracy, consistency, and reliability of data pipelines and data-driven systems.

The ideal candidate will have strong hands-on experience in Data Testing, ETL/ELT Testing,

Database Testing, SQL, Python, Apache Spark, and Databricks. The candidate should be comfortable validating complex data transformations, investigating data discrepancies, and testing end-to-end data workflows across multiple sources, processing layers, and downstream systems.

You will work closely with Data Engineers, Software Engineers, Database Architects, Data Analysts, Data Scientists, and Product teams to ensure data is processed correctly and delivered reliably to downstream consumers.

This role requires strong analytical and problem-solving skills, attention to detail, and the ability to independently understand complex data flows, develop comprehensive test strategies, automate data validation, investigate defects, and drive quality across the data engineering ecosystem.

Key Responsibilities

• Design and execute functional, integration, regression, and end-to-end testing for data pipelines and data platforms.
• Validate ETL/ELT workflows, data transformations, business rules, and source-to-target data accuracy.
• Test Apache Spark/PySpark, Databricks, and Delta Lake data processing workflows.
• Perform advanced SQL-based data validation, reconciliation, and data-quality testing.
• Validate data for accuracy, completeness, consistency, uniqueness, integrity, and schema compliance.
• Test APIs and integrations across data sources, databases, pipelines, and downstream systems.
• Develop and maintain Python/SQL-based test automation for data validation and regression testing.
• Design and execute positive, negative, edge-case, failure, retry, and recovery scenarios across data workflows.
• Investigate data discrepancies and work with Data Engineers, Developers, Analysts, and Data Scientists to identify root causes and resolve issues.
• Review requirements, technical designs, and data flows to identify QA risks, dependencies, and coverage gaps early in the development lifecycle.
• Maintain comprehensive test cases, automation suites, defect documentation, test evidence, and release validation for assigned data products.
• Drive continuous improvement of QA processes, data validation frameworks, automation coverage, and overall data quality.

Required Qualifications & Skills

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
• 5+ years of relevant QA/testing experience, preferably working with Data Engineering, Big Data, or data-intensive applications.
• Strong hands-on experience in ETL/ELT testing, database testing, and data validation.
• Advanced SQL skills, including:
• Complex joins
• Subqueries
• CTEs
• Window functions
• Aggregations
• Data reconciliation
• Practical experience testing Apache Spark/PySpark data processing workflows.
• Hands-on experience with Databricks and Delta Lake.
• Strong understanding of Big Data concepts, distributed data processing, data pipelines, and data architecture.
• Experience validating Parquet, JSON, CSV, and other structured and semi-structured data formats.
• Strong Python programming/scripting skills for test automation and data validation.
• Experience with API and integration testing.
• Working knowledge of Linux/Unix environments and shell scripting.
• Understanding of:
• Data Lakes
• Lakehouse Architecture
• Data Warehousing
• ETL/ELT Pipelines
• Experience with Git, defect management tools, and CI/CD-based testing workflows.
• Experience working in an Agile/Scrum environment.
• Strong analytical, troubleshooting, and root-cause analysis skills.
• Ability to work independently, take ownership, and collaborate effectively with cross-functional teams.
• Excellent written and verbal communication skills in English.

Core Technology Areas

Data Testing | ETL/ELT | SQL | Python | Apache Spark | PySpark | Databricks | Delta Lake | Database Testing | API Testing | Data Quality | Data Lakes | Lakehouse | Data Warehousing | Linux/Unix | Shell Scripting | Git | CI/CD