{"id":4376336,"url":"https://alion.io/job/weekday-lead-data-engineer","title":"Lead Data Engineer","company":{"id":7097,"name":"Weekday","domain":"weekday.works","url":"https://alion.io/company/weekday","size_band":"51-200","is_staffing_agency":true,"employer_type":"agency","is_intermediary":false,"listed_via":null,"ats_vendor":"Workable","truth_index":{"grade":"B","score":80,"open_postings":123,"ghost_share":0,"stale_share":0.984,"repost_share":0,"time_to_fill_p50_days":5,"computed_at":"2026-11-16T05:46:25Z"}},"role":"Data Science","role_family":"Data Science","seniority":"lead","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"explicit","locations":["Delhi, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Airflow","optional":false},{"name":"Amazon Redshift","optional":false},{"name":"Apache Kafka","optional":false},{"name":"Apache Pulsar","optional":false},{"name":"AWS Glue","optional":false},{"name":"Azure Data Factory","optional":false},{"name":"CI/CD","optional":false},{"name":"Docker","optional":false},{"name":"Flink","optional":false},{"name":"Google BigQuery","optional":false},{"name":"Hadoop","optional":false},{"name":"Java","optional":false},{"name":"Kubernetes","optional":false},{"name":"Python","optional":false},{"name":"RabbitMQ","optional":false},{"name":"Scala","optional":false},{"name":"Snowflake","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false}],"status":"live","first_seen_at":"2026-11-16T14:58:35Z","employer_posted_date":"2026-11-16","last_verified_at":"2026-11-16T14:58:35Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"'\n: - ( - )\nExperience: 6+ yrs\nLocation: NCR, Delhi, India\nAs a Lead Data Engineer, you will spearhead the design, development, and optimization of large-scale data pipelines that support both batch and real-time data processing initiatives across the organization. You will collaborate closely with data scientists, product managers, and infrastructure teams to ensure data integrity, scalability, and high performance at every stage of the pipeline, while driving continuous improvement through automation, monitoring, and best practices. Your leadership will shape the data architecture, protect data quality, and enable timely, actionable insights that inform business decisions. In a fast-moving environment, you will balance technical excellence with strategic vision, fostering a culture of collaboration and innovation within the data engineering team.\nRequirements\nDesign and implement robust, end-to-end batch data pipelines using tools such as Apache Spark, Hadoop, and Airflow.\nDevelop real-time streaming pipelines with technologies like Apache Kafka, Flink, or Spark Structured Streaming to deliver low-latency analytics.\nOptimize pipeline performance through partitioning, indexing, and efficient data format selection (e.g., Parquet, ORC).\nEstablish data governance, quality checks, and automated monitoring to ensure data reliability and compliance.\nLead capacity planning and scaling strategies for data infrastructure, including cluster management and cost optimization.\nMentor and coach junior data engineers, fostering a collaborative and growth-oriented team culture.\nCollaborate with Data Scientists to translate analytical requirements into scalable pipeline solutions.\nDrive integration of data pipelines with downstream applications, dashboards, and reporting tools.\nStay current with emerging data engineering technologies and evaluate their applicability to business needs.\nParticipate in incident response and root cause analysis to quickly resolve data pipeline failures.\n6+ years of experience in data engineering with a proven track record in both batch and real-time processing.\nDeep knowledge of distributed computing frameworks such as Hadoop, Spark, and Flink.\nProficiency in message queuing and stream processing systems like Kafka, Pulsar, or RabbitMQ.\nStrong scripting skills in Python, Scala, or Java, and experience with SQL on large datasets.\nHands-on experience with cloud data services (AWS Glue, Redshift, BigQuery, Snowflake, or Azure Data Factory).\nFamiliarity with containerization (Docker, Kubernetes) and CI/CD pipelines for data workloads.\nDemonstrated ability to architect and scale data pipelines to handle petabyte-scale volumes.\nExcellent communication skills, capable of translating technical concepts for non-technical stakeholders.\nA proactive problem-solver with a passion for data quality, governance, and automation.\nLeadership aptitude, with experience building, mentoring, and growing data engineering teams.","description_format":"text","description_chars":2971,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"India","iso":"IN","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-11-16T14:58:35Z"}],"visa":[],"liveness":{"score":52,"band":"ok","label":"Likely open","p_open":1,"p_active":0.516,"p_room":1,"age_days":0,"expected_fill_days":5,"reasons":["conf:1","agency","win:early","comp:brand"],"computed_at":"2026-11-16T16:03:32Z"},"pay":null,"html_url":"https://alion.io/job/weekday-lead-data-engineer","json_url":"https://alion.io/job/weekday-lead-data-engineer.json","meta":{"generated_at":"2026-11-16T16:03:32Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":25612,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-17T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":7097},"rest":"https://alion.io/mcp/rest/get_company?id=7097"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fweekday-lead-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fweekday-lead-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fweekday-lead-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/weekday-lead-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fweekday-lead-data-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4376336}},{"tool":"search_jobs","arguments":{"company":"weekday.works","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4376336},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}