Data Quality Engineer

Responsibilities:

• Validate batch and streaming data pipelines for correctness, completeness, consistency, and timeliness.
• Create and maintain data quality checks (nulls, duplicates, schema drift, and referential integrity).
• Verify business rules and transformations using SQL-based validations.
• Design, develop, and executetest strategiesfor Databricks-based data pipelines and analytics workflows
• Establish data reconciliation and end-to-end traceability between source and downstream systems.
• Test ETL/ELT pipelines built using AWS services (Glue, Lambda, EMR, Step Functions).
• Validate transformations written in SQL and Python. Ensure correctness across data ingestion, enrichment, aggregation, and publishing layers.
• Test reprocessing, backfills, and historical data loads.
• Validate ETL/ELT processes built using Apache Spark (PySpark/Scala) in Databricks.
• Validate Kafka-based streaming pipelines for data integrity, ordering, and exactly-once/at-least-once semantics.
• Test producer and consumer logic and serialization formats (Avro, JSON, Protobuf).
• Validate topic configurations, partitions, offsets, retention policies, and schema changes.
• Simulate and test late arrivals, duplicate events, and consumer failures.
• Test data workflows using AWS S3 Glue, Lambda, Redshift, Athena, Kinesis, DynamoDB, or similar services.
• Validate IAM roles, permissions, and secure data access.
• Verify data lifecycle policies, encryption, and storage optimizations.
• Build and maintain automated data testing frameworks using Python.
• Develop reusable test utilities, fixtures, and synthetic datasets. Integrate data tests into CI/CD pipelines for pre-merge, scheduled, and post-deployment validation.
• Enable automated alerts for data quality failures.
• Validate pipeline performance for large-scale datasets.
• Test throughput, latency, and concurrency under peak workloads.
• Validate retry logic, error handling, idempotency, and recovery mechanisms.
• Perform soak, regression, and failover testing Validate data pipeline metrics, logs, and alerts using CloudWatch, Prometheus, Grafana, or equivalent tools.
• Partner with teams to define data SLAs and SLOs. Participate in incident response, root-cause analysis, and postmortems related to data quality issues.

Requirements:

• 7+ years of experience in QA, SDET, or data quality engineering roles.
• Strong hands-on experience with SQL for complex data validation and analysis.
• Proficiency in Python for test automation and data validation.
• Experience testing data pipelines and ETL/ELT workflows.
• Hands-on experience with Kafka or other streaming platforms.
• Solid understanding of AWS data services (S3 Glue, Redshift, Lambda, Athena, etc. ).
• Experience working with large datasets and distributed systems.
• Strong debugging, analytical, and problem-solving skills.

Skills

Python, QA, SDET, SQL, data quality