{"id":3201248,"url":"https://alion.io/job/intercontinental-exchange-senior-developer-data-engineer","title":"Senior Developer, Data Engineer","company":{"id":58612,"name":"Intercontinental Exchange","domain":"ice.com","url":"https://alion.io/company/intercontinental-exchange","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Jibe","truth_index":{"grade":"B","score":80,"open_postings":37,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":9,"computed_at":"2026-10-25T05:49:26Z"}},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Hyderabad, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":21000,"max_usd":43000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":71},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Airflow","optional":false},{"name":"Amazon S3","optional":false},{"name":"Anomaly Detection","optional":false},{"name":"Apache Iceberg","optional":false},{"name":"Apache Kafka","optional":false},{"name":"AWS","optional":false},{"name":"Azure","optional":false},{"name":"Collibra","optional":false},{"name":"Databricks","optional":false},{"name":"dbt","optional":false},{"name":"Delta Lake","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Feature Store","optional":false},{"name":"Flink","optional":false},{"name":"GCP","optional":false},{"name":"Great Expectations","optional":false},{"name":"Helm","optional":false},{"name":"Kubernetes","optional":false},{"name":"Machine Learning","optional":false},{"name":"MLFlow","optional":false},{"name":"OpenShift","optional":false},{"name":"Pulumi","optional":false},{"name":"Pydantic","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Rest API","optional":false},{"name":"Spark","optional":false},{"name":"SQLAlchemy","optional":false},{"name":"Terraform","optional":false}],"status":"live","first_seen_at":"2026-10-01T07:02:00Z","employer_posted_date":"2026-10-24","last_verified_at":"2026-10-25T14:56:41Z","board_verified":true,"closed_at":null,"days_open":24,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":24},"description":"Job Purpose\nIntercontinental Exchange, Inc. (ICE) is seeking an experienced Senior Data Engineer to join the AI Centre of Excellence (AI CoE) team. The AI CoE drives the adoption of artificial intelligence and advanced analytics across ICE's global exchange and financial data business, bringing together data engineering, machine learning, and applied research to deliver intelligent solutions at scale. This role sits at the foundation of that work, ensuring the data infrastructure, pipelines, and platforms the AI CoE depends on are robust, scalable, and fit for purpose.\nThe successful candidate will take ownership of the orchestration layer that keeps our data moving reliably, contribute to the lakehouse architecture that provides clean and trusted data to analytical and ML consumers, and develop the transformation logic that converts raw inputs into high-quality, governed data products. The role requires close collaboration with data scientists, ML engineers, and business analysts, and carries meaningful input into the tooling decisions and engineering standards the team adopts.\nWe are looking for a candidate with demonstrable, production-scale experience in data engineering and pipeline orchestration. Technical depth is essential, as is the ability to produce well-structured, maintainable code and to operate effectively within a collaborative engineering team.\nResponsibilities\nOwn Apache Airflow end-to-end. You will write DAGs that handle complex multi-step workflows across ingestion, transformation, validation, and delivery. That means getting the fundamentals right: idempotency, backfill safety, SLA alerting, dynamic task mapping, and thorough testing. You will also review other engineers' DAGs and raise the standard across the team.\nKeep Airflow running well on Kubernetes. You will manage the deployment using KubernetesExecutor or CeleryKubernetesExecutor, handle Helm upgrades, tune pod templates and resource limits, and deal with scheduler performance issues before they become incidents.\nBuild ETL and ELT pipelines that hold up in production. Sources will vary: APIs, message queues, databases, object storage. The expectation is that pipelines are fault-tolerant, incremental where it makes sense, and straightforward to debug when something goes wrong.\nTreat data quality as part of the job, not an afterthought. Schema validation, row-count reconciliation, and anomaly detection should be baked into pipelines from the start. Freshness and accuracy are part of your definition of done.\nDesign and maintain a lakehouse architecture using Apache Iceberg as the primary table format, following a Bronze, Silver, and Gold medallion structure. Schema evolution, partitioning strategies, and time-travel queries will be regular concerns, not edge cases.\nUse Databricks to manage cluster compute, Delta Lake workflows, Unity Catalog, and scheduled jobs. You will keep things cost-efficient, well-organised, and easy for others to navigate.\nBuild and maintain dbt projects that sit alongside the Airflow orchestration layer. Models, tests, snapshots, and macros should be well-structured and documented. Test failures and freshness checks should surface where people can act on them.\nContribute reusable components to the shared platform: pipeline templates, custom Airflow operators, and utility libraries that make the whole team faster and more consistent.\nPartner with data scientists and ML engineers to deliver feature pipelines and training datasets with the freshness and reproducibility their models need.\nInstrument your work. Pipelines should have metrics, dashboards, and alerts that surface real problems early. You will own runbooks and participate in on-call cover.\nPlay an active part in architecture discussions, tooling evaluations, and mentoring. Your experience should benefit the team, not just your own workstream.\nKnowledge and Experience\nApache Airflow: Expert level, with at least three years running Airflow in production. You should know the scheduler internals, executor types, XComs, Connections, Variables, pools, and the TaskFlow API well enough to make trade-off decisions and explain them clearly.\nAirflow on Kubernetes: Real experience deploying and operating Airflow on Kubernetes via KubernetesExecutor or CeleryKubernetesExecutor: pod templates, resource tuning, persistent volume claims, Helm chart management, and upgrades that do not take the scheduler offline.\nDAG Development: Strong Python applied to DAG authoring: dynamic task mapping, custom operators and sensors, cross-DAG triggers, parameterised pipelines, and proper test coverage. Writing testable DAGs should feel natural, not optional.\nETL and ELT Pipelines: Demonstrated experience building batch and incremental pipelines at scale. You understand transformation logic, data quality validation, and lineage capture, and you have shipped pipelines that other people maintain without you.\nData Lakehouse Architecture: A solid grasp of lakehouse design in practice: how to implement a medallion model, when to use which table format, and how to make data reliably available to downstream consumers without constant hand-holding.\nApache Iceberg: Hands-on knowledge of the Iceberg table spec: snapshot isolation, partition evolution, hidden partitioning, and time-travel queries. You have managed Iceberg tables in a real catalog environment, whether Hive metastore, a REST catalog, or otherwise.\nDatabricks: Practical Databricks experience covering cluster management, Delta Lake, Workflows or Jobs, and Unity Catalog. Experience with Databricks Asset Bundles or MLflow is a welcome bonus.\ndbt: Confident building dbt projects from the ground up: models, sources, tests, snapshots, seeds, and macros. You know how to wire dbt into Airflow, write documentation that people actually use, and make freshness checks meaningful.\nPython: Strong engineering-grade Python: well-structured, tested, and packaged. Comfortable with pandas, PySpark, SQLAlchemy, and Pydantic. You write code other engineers want to read.\nPreferred Knowledge and Experience\nApache Kafka and Streaming: Experience with near-real-time ingestion using Kafka, Kafka Connect, or Flink, including landing data into lakehouse storage.\nApache Spark: PySpark proficiency for large-scale distributed processing, with a feel for memory management, shuffle optimisation, and partition sizing.\nKubernetes and OpenShift: Platform knowledge beyond Airflow: Helm, RBAC, and namespace management, enough to self-serve on infrastructure without leaning on platform teams.\nData Governance and Contracts: Familiarity with data contract approaches, catalogue tools such as DataHub or Collibra, and quality frameworks like Great Expectations or Soda.\nCloud Platforms: Experience with managed data services on AWS, Azure, or GCP: S3, Glue, ADLS, Synapse, GCS, Dataproc, and similar.\nInfrastructure as Code: Ability to provision data infrastructure using Terraform or Pulumi.\nML Feature Engineering: An understanding of feature stores and experience building pipelines that serve model training and inference reliably.","description_format":"text","description_chars":7100,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Stock Exchanges & Market Infrastructure"],"lifecycle":[{"event":"open","at":"2026-10-24T08:18:53Z"}],"visa":[],"liveness":{"score":13,"band":"cold","label":"Long shot","p_open":1,"p_active":0.383,"p_room":0.35,"age_days":23,"expected_fill_days":9,"reasons":["conf:2","stale_co","win:tail","comp:brand"],"computed_at":"2026-10-25T05:49:26Z"},"pay":null,"html_url":"https://alion.io/job/intercontinental-exchange-senior-developer-data-engineer","json_url":"https://alion.io/job/intercontinental-exchange-senior-developer-data-engineer.json","meta":{"generated_at":"2026-10-25T17:44:52Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":25346,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-26T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":58612},"rest":"https://alion.io/mcp/rest/get_company?id=58612"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fintercontinental-exchange-senior-developer-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fintercontinental-exchange-senior-developer-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fintercontinental-exchange-senior-developer-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/intercontinental-exchange-senior-developer-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fintercontinental-exchange-senior-developer-data-engineer"}]}