{"id":4434177,"url":"https://alion.io/job/go-digital-technology-consulting-lead-data-engineer","title":"Lead Data Engineer","company":{"id":4513157,"name":"Go Digital Technology Consulting","domain":"godigitaltc.com","url":"https://alion.io/company/go-digital-technology-consulting","size_band":null,"is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":"Keka","truth_index":{"grade":"B","score":75,"open_postings":5,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-11-17T05:47:56Z"}},"role":"Data Science","role_family":"Data Science","seniority":"lead","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Mumbai, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":4,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"Amazon Kinesis","optional":false},{"name":"Amazon Redshift","optional":false},{"name":"Apache Kafka","optional":false},{"name":"AWS","optional":false},{"name":"AWS Lambda","optional":false},{"name":"AWS Step Functions","optional":false},{"name":"CI/CD","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Git","optional":false},{"name":"Hadoop","optional":false},{"name":"NumPy","optional":false},{"name":"Pandas","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false}],"status":"live","first_seen_at":"2026-04-20T04:47:26Z","employer_posted_date":"2026-04-20","last_verified_at":"2026-11-17T04:48:05Z","board_verified":true,"closed_at":null,"days_open":211,"trust":{"level":"stale","repost_count":0,"flags":["stale"],"days_open":211},"description":"Technologies / Skills:\nAdvanced SQL, Python and associated libraries like Pandas, Numpy etc., Pyspark , Shell scripting, Data\u0002Modelling, Big data, Hadoop , Hive, ETL pipelines and IaC tools like Terraform etc.\nResponsibilities:\nProven success in communicating with users, other technical teams, and senior management to \nCollect requirements, describe data modeling decisions and develop data engineering strategy.\nAbility to work with business owners to define key business requirements and convert to user stories with required technical specifications.\nCommunicate results and business impacts of insight initiatives to key stakeholders to collaboratively solve business problems.\nWorking closely with the overall Enterprise Data & Analytics Architect and Engineering practice leads to ensure adherence with the best practices and design principles.\nAssures quality, security and compliance requirements are met for supported area.\nDesign and create fault-tolerance data pipelines running on cluster\nExcellent communication skills with the ability to influence client business and IT teams\nShould have design data engineering solutions end to end. Ability to come up with scalable and modular solutions\nRequired Qualification:\n7 years of relevant IT experience\n4+ years of hands-on experience Designing and developing Data Pipelines for Data Ingestion or Transformation using Python (PySpark)/Spark SQL in AWS cloud\nExperience in design and development of data pipelines and processing of data at scale using technologies like EMR, Kinesis, Kafka, Lambda, Glue, Athena, Redshift, Step Functions.\nAdvanced experience in writing and optimizing efficient SQL queries with Python and Hive handling Large Data Sets in Big-Data Environments\nExperience in debugging, tunning and optimizing PySpark data pipelines\nShould have implemented concepts and have good knowledge of Pyspark dataframes, joins, caching, memory management, partitioning, parallelism etc.\nUnderstanding of Spark UI, Event Timelines, DAG, Spark config parameters, in order to tune the long running data pipelines. \nExperience working in Agile implementations\nExperience with building data pipelines in streaming and batch mode.\nExperience with Git and CI/CD pipelines to deploy cloud applications\nGood knowledge of designing Hive tables with partitioning for performance\nDesired Qualification:\nExperience in data modelling\nHands on creating workflows on any Scheduling Tool like Autosys, CA Workload Automation\nProficiency in using SDKs for interacting with native AWS services\nStrong understanding of concepts of ETL, ELT and data modeling","description_format":"text","description_chars":2605,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation"],"lifecycle":[{"event":"open","at":"2026-11-17T04:48:05Z"}],"visa":[],"liveness":{"score":7,"band":"cold","label":"Long shot","p_open":1,"p_active":0.246,"p_room":0.28,"age_days":211,"expected_fill_days":29,"reasons":["conf:0","win:tail","crowd:"],"computed_at":"2026-11-17T05:47:56Z"},"pay":null,"html_url":"https://alion.io/job/go-digital-technology-consulting-lead-data-engineer","json_url":"https://alion.io/job/go-digital-technology-consulting-lead-data-engineer.json","meta":{"generated_at":"2026-11-17T06:02:39Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":11102,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":4513157},"rest":"https://alion.io/mcp/rest/get_company?id=4513157"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fgo-digital-technology-consulting-lead-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fgo-digital-technology-consulting-lead-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fgo-digital-technology-consulting-lead-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/go-digital-technology-consulting-lead-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fgo-digital-technology-consulting-lead-data-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4434177}},{"tool":"search_jobs","arguments":{"company":"godigitaltc.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4434177},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}