{"id":4613264,"url":"https://alion.io/job/sampoorna-consultants-data-engineer-aws-data-pipelines","title":"Data Engineer - AWS Data Pipelines","company":{"id":3800140,"name":"Sampoorna Consultants","domain":"sampoorna.com","url":"https://alion.io/company/sampoorna-computer-people","size_band":"51-200","is_staffing_agency":true,"employer_type":"agency","is_intermediary":false,"listed_via":null,"ats_vendor":"Schema","truth_index":{"grade":"A","score":97,"open_postings":11,"ghost_share":0,"stale_share":0,"repost_share":0.182,"time_to_fill_p50_days":41,"computed_at":"2026-11-19T09:08:57Z"}},"role":"Data Science","role_family":"Data Science","seniority":null,"employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Airflow","optional":false},{"name":"Amazon S3","optional":false},{"name":"AWS","optional":false},{"name":"AWS Glue","optional":false},{"name":"dbt","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Python","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false},{"name":"AI Agents","optional":true},{"name":"Apache Kafka","optional":true},{"name":"Hadoop","optional":true},{"name":"LLM","optional":true},{"name":"Machine Learning","optional":true},{"name":"pySpark","optional":true},{"name":"Spark","optional":true}],"status":"live","first_seen_at":"2026-11-19T15:45:07Z","employer_posted_date":"2026-11-19","last_verified_at":"2026-11-19T15:45:07Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"1.Mandatory Technical Skills - Must Have\nA. Apache Airflow - Self-Managed (Critical Requirement)\nHands-on experience installing, configuring, deploying, and operating Apache Airflow.\nExperience managing DAGs, scheduling, dependencies, retries, monitoring, troubleshooting, and production failures.\nUnderstanding of Airflow infrastructure, workers, schedulers, executors, and operational maintenance.\nExperience running Airflow independently, not exclusively through managed platforms such as AWS MWAA or Astronomer.\nImportant: Candidates who have only created or maintained DAGs within an existing managed Airflow environment are not an exact match.\nB. AWS Data Engineering\nHands-on experience building and maintaining data pipelines using:\nAmazon S3: Data storage, ingestion, and organization.\nAWS Glue: ETL jobs, Glue Data Catalog, and data transformations.\nAmazon Athena: SQL-based querying, validation, and analysis of data stored in S3.\nCandidates should demonstrate how these services were used together in actual production pipelines.\nC. SQL & Python - Strong Hands-On Expertise\nAdvanced SQL skills, including joins, CTEs, window functions, aggregations, and query optimization.\nStrong Python programming skills for building ETL/ELT pipelines, transformations, automation, and data validation.\nExperience troubleshooting data issues and improving pipeline reliability.\nD. Data Quality & Reconciliation - Critical Requirement\nMust have experience ensuring the accuracy and integrity of data processed through pipelines.\nRelevant experience includes:\nSource-to-target data reconciliation.\nRecord count and aggregate validation.\nIdentifying missing, duplicate, inconsistent, or incorrect records.\nInvestigating data mismatches and transformation errors.\nImplementing automated data quality checks.\nEnsuring business metrics and reporting outputs are accurate.\nImportant: Candidates focused only on pipeline execution, scheduling, or infrastructure monitoring without data validation experience are not a strong match.\nE. dbt (Data Build Tool)\nHands-on experience developing SQL transformation models using dbt.\nExperience with dbt tests, dependencies, documentation, and data transformation workflows.\nAbility to demonstrate dbt usage in actual projects.\nF. Terraform - Infrastructure as Code\nPractical experience using Terraform to provision and manage cloud infrastructure.\nExperience maintaining Terraform configurations and deploying AWS resources.\nUnderstanding of infrastructure version control and repeatable deployments.\n3. Preferred / Good-to-Have Experience\nExperience working with healthcare, health-tech, or personal/sensitive data.\nFamiliarity with data privacy, security, and compliance requirements.\nExperience handling regulated or confidential datasets.\nExperience troubleshooting and maintaining production data pipelines.\nNOTE : Profiles to Avoid / Deprioritize\nPlease avoid submitting candidates whose primary experience is:\nAI/GenAI/LLM Engineering: Primarily focused on AI agents, machine learning, or GenAI applications rather than traditional data engineering.\nLarge-Scale Big Data Engineering: Primarily focused on Spark, PySpark, Kafka, Hadoop, or streaming platforms without strong experience in the required AWS stack.\nManaged Airflow Only: Experience limited to using AWS MWAA, Astronomer, or an already configured Airflow environment.\nETL Tool Operators: Experience limited to running or monitoring pipelines without developing, troubleshooting, or validating data.\nSkills-Only Resumes: Candidates listing the required technologies without demonstrating hands-on project experience.\nSpark, Kafka, and AI experience are not disqualifying, but they should not substitute for the mandatory skills.\n2. Mandatory Resume Submission Guidelines :\nBefore sharing a profile with the client, please ensure:\n1.Employment and Project Dates: Include start and end dates (month and year) for each role and relevant project.\n2.Project-Based Technical Evidence: Clearly describe how Airflow, AWS S3, Athena, Glue, SQL, Python, dbt, and Terraform were used in actual projects.\n3.Airflow Ownership: Explicitly mention whether the candidate independently deployed and managed Airflow or only worked with an existing managed environment.\n4.Data Quality Experience: Highlight specific examples of data reconciliation, validation, accuracy checks, and issue resolution.\n5.Previous Application Status: Confirm whether the candidate has previously applied, interviewed, or been submitted to this client.","description_format":"text","description_chars":4512,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-11-19T15:45:07Z"}],"visa":[],"liveness":{"score":52,"band":"ok","label":"Likely open","p_open":1,"p_active":0.516,"p_room":1,"age_days":0,"expected_fill_days":41,"reasons":["conf:0","agency","win:early"],"computed_at":"2026-11-19T16:38:05Z"},"pay":null,"html_url":"https://alion.io/job/sampoorna-consultants-data-engineer-aws-data-pipelines","json_url":"https://alion.io/job/sampoorna-consultants-data-engineer-aws-data-pipelines.json","meta":{"generated_at":"2026-11-19T16:38:05Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":27628,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-20T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3800140},"rest":"https://alion.io/mcp/rest/get_company?id=3800140"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fsampoorna-consultants-data-engineer-aws-data-pipelines"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fsampoorna-consultants-data-engineer-aws-data-pipelines"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fsampoorna-consultants-data-engineer-aws-data-pipelines"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/sampoorna-consultants-data-engineer-aws-data-pipelines\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fsampoorna-consultants-data-engineer-aws-data-pipelines"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4613264}},{"tool":"search_jobs","arguments":{"company":"sampoorna.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4613264},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}