{"id":2780022,"url":"https://alion.io/job/relanto-lead-data-engineer","title":"Lead Data Engineer","company":{"id":4442664,"name":"Relanto","domain":"relanto.ai","url":"https://alion.io/company/relanto-2","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Keka","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"lead","employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":30000,"max_usd":53000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":16},"experience_years_min":6,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Airflow","optional":false},{"name":"Apache Kafka","optional":false},{"name":"CI/CD","optional":false},{"name":"ClickHouse","optional":false},{"name":"Flink","optional":false},{"name":"Git","optional":false},{"name":"Java","optional":false},{"name":"Python","optional":false},{"name":"Scala","optional":false},{"name":"SQL","optional":false},{"name":"Windows","optional":false},{"name":"Apache NiFi","optional":true},{"name":"AWS","optional":true},{"name":"Azure","optional":true},{"name":"Dagster","optional":true},{"name":"Docker","optional":true},{"name":"GCP","optional":true},{"name":"Kubernetes","optional":true},{"name":"Prefect","optional":true},{"name":"Rest API","optional":true},{"name":"Terraform","optional":true}],"status":"live","first_seen_at":"2026-09-29T14:05:13Z","employer_posted_date":"2026-09-29","last_verified_at":"2026-10-17T22:12:39Z","board_verified":true,"closed_at":null,"days_open":18,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":18},"description":"Role: Lead Data Engineer\nExperience: 6-8 years\nLocation: Bengaluru / Hybrid\nEmployment: Full-time\nRole Summary\nWe are looking for a Lead Data Engineer with strong hands-on experience in real-time data streaming, event processing, CDC, data integration, and modern data engineering.\nThe Lead Data Engineer will be responsible for designing, developing, and maintaining highperformance data pipelines using Apache Flink, Apache Kafka, Debezium, CDC, ClickHouse, and Apache Airflow.\nThis is a hands-on engineering role requiring strong practical experience in building productiongrade streaming applications, developing data pipelines, troubleshooting distributed systems, and optimizing data processing workloads.\nThe ideal candidate should be comfortable working across the full data pipeline-from source systems and CDC ingestion through Kafka and Flink processing to analytical storage, APIs, dashboards, and downstream integrations.\nKey Responsibilities\n Real-Time Data Engineering\nDesign, develop, and maintain real-time data pipelines using Apache Flink and Apache Kafka. \n Develop production-grade streaming applications for high-volume and low-latency workloads.\nImplement data transformation, filtering, enrichment, aggregation, and event processing. \nBuild reliable event-processing pipelines with appropriate error handling and recovery mechanisms. \nConsume and publish events across Kafka topics. • Implement appropriate partitioning, consumer groups, offsets, and delivery mechanisms.\nTroubleshoot streaming pipeline failures and performance issues. \nApache Flink\nDevelop and maintain Apache Flink jobs forreal-time data processing.\nImplement:Stream transformations\nFiltering\nMapping\nAggregations\nJoins\nWindows\nEvent-time processing\nWatermarks\nState management\n\nImplement Flink checkpointing and recovery mechanisms.\nOptimize Flink jobs for performance, scalability, and resource utilization.\nMonitor Flink jobs for latency, throughput, failures, and resource consumption.\nTroubleshoot state, checkpointing, backpressure, and processing issues.\nKafka\nDevelop Kafka-based ingestion and streaming pipelines.\nCreate and manage Kafka topics and event streams.\nWorkwith partitions, offsets,consumer groups, replication, and retention.\nImplement reliableproducer and consumer applications.\nHandle message ordering, retries,duplicate events, and replay scenarios.\nMonitor Kafka performance and troubleshoot consumerlag and throughput issues.\nWork with Kafka schemasand serialization formats.\nCDC & Debezium\nBuildCDC-based ingestion pipelines using Debezium.\nConfigure and maintain Debeziumconnectors.\nCapture source-system inserts, updates,and deletes.\nPublish CDC events into Kafka.\nHandle initial snapshots and incremental CDC processing.\nManage schemaevolution and changesin source systems.\nImplement data reconciliation and consistency checks.\nTroubleshoot CDC failures and source-to-target data issues.\nData Orchestration\nDevelop and maintain data workflows using Apache Airflow orequivalent orchestration frameworks.\nBuildreusable DAGs for:Data ingestion\nCDC workflows\nData validation\nFlink job execution\nData transformation\nClickHouse loading\nDownstream integrations\n\nImplement workflowdependencies, scheduling, retries,backfills, SLAs, and alerting.\nIntegrate Airflow workflows with Kafka, Flink, Debezium, ClickHouse, APIs, and cloud services.\nMonitor workflow execution and troubleshoot failures.\nDevelop reusable operators, sensors,and workflow components where appropriate.\nUseevent-driven triggers where real-time workflows require them.\nClickHouse & Analytical Data\nIntegrate streaming data pipelines with ClickHouse.\nDesign efficient analytical data models.\nDevelop and optimize SQL queries.\nImplement appropriate partitioning, sorting, indexing,and retention strategies.\nOptimize data ingestion and query performance.\nSupport downstream analytical use cases,dashboards, and reporting requirements.\nData Quality& Reliability\nImplement data validation and quality checksthroughout the pipeline.\nBuildreconciliation mechanisms betweensource and target systems.\nMonitor data freshness, completeness, accuracy, and consistency.\nImplement error handling, retry,replay, and recoverymechanisms.\nEstablish loggingand observability for critical pipelines.\nSupport incident investigation and root-cause analysis.\nIntegration & APIs\nIntegrate streaming and analytical data with APIs, endpoints, dashboards, and downstream applications.\nDevelop data interfaces and integration components.\nWorkwith application teams to define data contractsand integration requirements.\nSupport future integrations and additional data consumers.\nEngineering Practices\nFollow modernsoftware engineering practices around:Git\nCode reviews\nUnit testing\nIntegration testing\nCI/CD\nLogging\nMonitoring\nDocumentation\n\nDevelop reusable and maintainable data engineering components.\nParticipate in technical designdiscussions and architecture reviews.\nMentor other Data Engineers and contribute to engineering standards.\nRequired Skills& Experience\n6-8 years ofexperience in Data Engineering.\nStrong hands-onexperience with Apache Flink - mandatory/core requirement.\nStrong hands-on experience with Apache Kafka.\nHands-on experience with Debeziumand CDC.\nStrong programming experience in Java or Scala.\nPython experience is an advantage.\nStrong SQL skills.\nExperience with analytical databases; ClickHouse is highly preferred.\nHands-on experience with Apache Airflowor another data orchestration framework.\nStrong understanding of distributed systemsand real-time data processing.\nExperience developing and supporting production-grade streaming pipelines.\nStrong understanding of Kafka:Topics\nPartitions\nOffsets\nConsumer groups\nReplication\nRetention\n\nExperience with data transformation, enrichment, filtering, aggregation, and event processing.\nExperience troubleshooting performance and reliability issues.\nFamiliarity with cloud platforms and containerized environments.\nPreferred Skills\nAdvanced experience with Apache Flink, including:State management\nCheckpoints\nSavepoints\nWatermarks\nEvent time\nWindows\nBackpressure\nState backends\n\nExperience with Apache Airflow, Dagster,Prefect, or Apache NiFi.\nExperience with Kafka Schema Registry.\nExperience with Avro, Protobuf, or JSON.\nExperience with Kubernetes.\nExperience with AWS / Azure / GCP.\nExperience with CI/CD pipelines.\nExperience with Docker and containerized applications.\nExperience with Terraform or other Infrastructure as Code tools.\nExperience with data observability and monitoring tools.\nExperience building high-volume, low-latency real-time data platforms.\nExperience with REST APIs and systemintegrations.","description_format":"text","description_chars":6656,"description_truncated":false,"requirements":{"experience_years_min":6,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"India","iso":"IN","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-10-17T12:17:51Z"}],"visa":[],"liveness":{"score":57,"band":"ok","label":"Likely open","p_open":1,"p_active":0.761,"p_room":0.75,"age_days":18,"expected_fill_days":26,"reasons":["conf:2","win:late"],"computed_at":"2026-10-18T00:22:17Z"},"pay":null,"html_url":"https://alion.io/job/relanto-lead-data-engineer","json_url":"https://alion.io/job/relanto-lead-data-engineer.json","meta":{"generated_at":"2026-10-18T00:22:17Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":657,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-19T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":4442664},"rest":"https://alion.io/mcp/rest/get_company?id=4442664"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Frelanto-lead-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Frelanto-lead-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Frelanto-lead-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/relanto-lead-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Frelanto-lead-data-engineer"}]}