{"id":2771137,"url":"https://alion.io/job/qentelli-data-engineer-pythonpyspark","title":"Data Engineer - Python/PySpark","company":{"id":1798376,"name":"Qentelli","domain":"qentelli.com","url":"https://alion.io/company/qentelli","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Hyderabad, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":21000,"max_usd":43000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":71},"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Amazon CloudWatch","optional":false},{"name":"Amazon Kinesis","optional":false},{"name":"Amazon Redshift","optional":false},{"name":"Amazon S3","optional":false},{"name":"Apache Kafka","optional":false},{"name":"AWS","optional":false},{"name":"AWS Glue","optional":false},{"name":"AWS Lambda","optional":false},{"name":"AWS Step Functions","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"DynamoDB","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Grafana","optional":false},{"name":"IAM","optional":false},{"name":"Prometheus","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Scala","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false}],"status":"live","first_seen_at":"2026-09-03T06:29:25Z","employer_posted_date":null,"last_verified_at":"2026-09-03T06:29:25Z","board_verified":false,"closed_at":null,"days_open":51,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":51},"description":"Job Description :\n\nWe are seeking a Data Engineer to ensure the reliability, accuracy, performance, and scalability of our data platforms and pipelines. This role focuses on validating end-to-end data ingestion, processing, streaming, and analytics workflows built on AWS, Kafka, SQL, Data Bricks and Python. You will work closely with Data Engineers, Platform Engineers, Analytics teams, and DevOps to embed quality into the data lifecycle ensuring trusted data, resilient pipelines, and production-ready systems.\n\nKey Responsibilities :\n\n- Validate batch and streaming data pipelines for correctness, completeness, consistency, and timeliness.\n\n- Create and maintain data quality checks (nulls, duplicates, schema drift, referential integrity).\n\n- Verify business rules and transformations using SQL-based validations.\n\n- Design, develop, and execute test strategies for Databricks-based data pipelines and analytics workflows.\n\n- Establish data reconciliation and end-to-end traceability between source and downstream systems.\n\n- Test ETL/ELT pipelines built using AWS services (Glue, Lambda, EMR, Step Functions).\n\n- Validate transformations written in SQL and Python.\n\n- Ensure correctness across data ingestion, enrichment, aggregation, and publishing layers.\n\n- Test reprocessing, backfills, and historical data loads.\n\n- Validate ETL/ELT processes built using Apache Spark (PySpark/Scala) in Databricks.\n\n- Validate Kafka-based streaming pipelines for data integrity, ordering, and exactly-once/at-least-once semantics.\n\n- Test producer and consumer logic, serialization formats (Avro, JSON, Protobuf).\n\n- Validate topic configurations, partitions, offsets, retention policies, and schema changes.\n\n- Simulate and test late arrivals, duplicate events, and consumer failures.\n\n- Test data workflows using AWS S3, Glue, Lambda, Redshift, Athena, Kinesis, DynamoDB, or similar services.\n\n- Validate IAM roles, permissions, and secure data access.\n\n- Verify data lifecycle policies, encryption, and storage optimizations.\n\n- Build and maintain automated data testing frameworks using Python.\n\n- Develop reusable test utilities, fixtures, and synthetic datasets.\n\n- Integrate data tests into CI/CD pipelines for pre-merge, scheduled, and post-deployment validation.\n\n- Enable automated alerts for data quality failures.\n\n- Validate pipeline performance for large-scale datasets.\n\n- Test throughput, latency, and concurrency under peak workloads.\n\n- Validate retry logic, error handling, idempotency, and recovery mechanisms.\n\n- Perform soak, regression, and failover testing.\n\n- Validate data pipeline metrics, logs, and alerts using CloudWatch, Prometheus, Grafana, or equivalent tools.\n\n- Partner with teams to define data SLAs and SLOs.\n\n- Participate in incident response, root-cause analysis, and postmortems related to data quality issues.\n\nRequired Qualifications :\n\n- 5+ years of experience in Data Quality Engineering roles.\n\n- Strong hands-on experience with SQL for complex data validation and analysis.\n\n- Proficiency in Python for test automation and data validation.\n\n- Experience testing data pipelines and ETL/ELT workflows.\n\n- Hands-on experience with Kafka or other streaming platforms.\n\n- Solid understanding of AWS data services (S3, Glue, Redshift, Lambda, Athena, etc.).\n\n- Experience working with large datasets and distributed systems.\n\n- Strong debugging, analytical, and problem-solving skills.\nSkills\nData Engineering, Python, PySpark, Databricks, SQL, AWS, ETL, AWS Glue, Kafka","description_format":"text","description_chars":3508,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Software Testing & QA","Custom Software Development"],"lifecycle":[{"event":"open","at":"2026-10-17T04:49:54Z"}],"visa":[],"liveness":{"score":11,"band":"cold","label":"Long shot","p_open":0.4,"p_active":0.596,"p_room":0.45,"age_days":50,"expected_fill_days":28,"reasons":["seen:50","velocity","win:tail"],"computed_at":"2026-10-24T06:02:35Z"},"pay":null,"html_url":"https://alion.io/job/qentelli-data-engineer-pythonpyspark","json_url":"https://alion.io/job/qentelli-data-engineer-pythonpyspark.json","meta":{"generated_at":"2026-10-24T15:11:44Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":21702,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-25T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1798376},"rest":"https://alion.io/mcp/rest/get_company?id=1798376"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fqentelli-data-engineer-pythonpyspark"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fqentelli-data-engineer-pythonpyspark"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fqentelli-data-engineer-pythonpyspark"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/qentelli-data-engineer-pythonpyspark\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fqentelli-data-engineer-pythonpyspark"}]}