{"id":3690369,"url":"https://alion.io/job/ntt-data-senior-data-engineer-4","title":"Senior Data Engineer","company":{"id":50348,"name":"NTT DATA","domain":"nttdata.com","url":"https://alion.io/company/ntt-data","size_band":"5000+","is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":"Phenom","truth_index":{"grade":"A","score":94,"open_postings":114,"ghost_share":0,"stale_share":0.412,"repost_share":0.026,"time_to_fill_p50_days":18,"computed_at":"2026-11-05T05:54:30Z"}},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":10,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Apache Iceberg","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Git","optional":false},{"name":"Hadoop","optional":false},{"name":"Linux","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Airflow","optional":true},{"name":"Amazon S3","optional":true},{"name":"CI/CD","optional":true},{"name":"Kubernetes","optional":true},{"name":"OpenShift","optional":true},{"name":"Polaris","optional":true},{"name":"PostgreSQL","optional":true},{"name":"Trino","optional":true}],"status":"live","first_seen_at":"2026-11-03T06:27:05Z","employer_posted_date":"2026-11-03","last_verified_at":"2026-11-05T13:27:48Z","board_verified":true,"closed_at":null,"days_open":2,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":2},"description":"Role Summary\nNTT DATA is seeking a highly experienced Senior Data Engineer / Technical Lead for a large-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg.\nThe role requires a strong hands-on engineer who can lead the engineering team while continuing to design, develop, troubleshoot and optimize the solution. The candidate will own technical execution of the migration framework, Spark/PySpark engineering, Iceberg implementation, data modelling/schema mapping, Cyber/TaaS integration, migration controls, reconciliation, performance tuning and technical delivery of migration waves.\nThis is not a coordination-only Technical Lead role. The candidate is expected to contribute to code, conduct design/code reviews, troubleshoot complex production issues and mentor Data Engineers.\nExperience\n10+ years of Data Engineering experience, including:\n5+ years hands-on Apache Spark/PySpark.\nStrong Hadoop/Hive experience.\nExperience leading Data Engineering teams.\nLarge-scale data migration/data modernization experience.\nStrong data modelling and source-to-target mapping experience.\nMulti-hundred-TB/PB-scale migration experience is strongly preferred.\nKey Responsibilities\nTechnical Leadership\nLead a team of Data Engineers through assessment, development, migration, validation and production execution.\nTranslate architecture and migration standards into detailed engineering designs and reusable implementation patterns.\nOwn technical estimates, engineering backlog, work allocation and technical delivery.\nConduct design and code reviews and enforce engineering standards.\nMentor Data Engineers and provide hands-on technical guidance.\nAct as the primary engineering escalation point for complex migration and production issues.\nPartner with Data Architect, Technical PM, security, infrastructure and PNC technical teams.\nLead root-cause analysis and ensure fixes are incorporated into reusable framework patterns rather than implemented as one-off solutions.\nHadoop-to-Iceberg Migration\nLead technical execution of approximately 2 PB of Hadoop/Hive historical-data migration to Apache Iceberg.\nHarden and extend the reusable, configuration-driven migration framework developed during the POC.\nAnalyze Hive/HDFS metadata, tables, partitions, file layouts and migration complexity.\nImplement migration strategies including bulk/CTAS, partition-wise and chunk-wise processing.\nDesign execution approaches for small, medium, large and very-large tables.\nOptimize parallel migration across tables and partitions.\nImplement checkpointing, restart/recovery, idempotency and safe retries.\nLead technical execution and troubleshooting of migration waves.\nData Modelling & Schema Engineering\nAnalyze existing Hive data models and translate them into approved Apache Iceberg physical models.\nDevelop and review conceptual/logical/physical data models where required, with primary emphasis on physical modelling for migration.\nPerform source-to-target data mapping and schema/data-type mapping.\nDefine and implement table structures consistent with approved Iceberg standards.\nAnalyze:\no entities and relationships,\no primary/foreign-key relationships,\no data types,\no partition keys,\no business keys,\no reference/cross-reference structures.\nIdentify schema incompatibilities and recommend remediation.\nWork with the Data Architect on partitioning, file-layout and model-design decisions.\nEnsure migrated models preserve required business relationships and downstream compatibility.\nSupport future Bronze/Silver/Gold modelling requirements where applicable.\nImportant: enterprise modelling standards remain owned by the Data Architect; the Senior DE/Technical Lead is responsible for detailed modelling and implementation within those standards.\nSpark/PySpark Engineering & Performance\nDevelop and review production-grade Python/PySpark components.\nTune Spark jobs for PB-scale migration.\nDiagnose Spark stages, executors, shuffle, skew, memory and I/O bottlenecks.\nOptimize table/partition/chunk parallelism.\nTune workloads to meet migration-throughput objectives.\nIdentify bottlenecks across:\nHadoop Read → Spark → TaaS/Tokenization → Iceberg Write → Reconciliation\nCoordinate technical remediation with infrastructure/storage/network teams where bottlenecks are outside the migration framework.\nApache Iceberg Engineering\nImplement approved Iceberg table standards.\nDevelop/configure Iceberg tables and Spark→Iceberg processing.\nImplement:\no partitioning,\no schema evolution,\no snapshots,\no file sizing,\no compaction,\no metadata maintenance,\no small-file remediation.\nTroubleshoot Iceberg catalog/write/commit issues.\nOptimize Iceberg table performance and storage characteristics.\nCyber-3 / Tokenization\nImplement PNC-approved TaaS/Protegrity tokenization and retokenization patterns.\nIntegrate tokenization into Spark migration pipelines.\nImplement batching, retry, timeout and recovery handling.\nTroubleshoot tokenization failures/performance bottlenecks.\nMeasure impact of tokenization on end-to-end migration throughput.\nSupport Cyber/tokenization validation based on PNC-defined rules.\nMigration Controls & Reconciliation\nLead implementation of PNC Data Movement Controls, including:\no checkpointing,\no idempotency,\no control totals,\no restart/recovery,\no retry,\no audit logging,\no exception handling.\nBuild reusable source-to-target reconciliation capabilities covering:\no row counts,\no schema validation,\no partition completeness,\no control totals,\no aggregates,\no hash/checksum where appropriate,\no null/duplicate validation,\no tokenization validation.\nIsolate discrepancies at table/partition level.\nEnable selective reprocessing rather than full-table reruns.\nProduce technical evidence required for migration acceptance.\nMandatory Skills\nStrong hands-on expertise in:\nPython\nPySpark\nApache Spark / Spark SQL\nHadoop / HDFS\nHive / HiveQL / Hive Metastore\nSQL\nApache Iceberg\nData modelling\nSource-to-target mapping\nParquet\nLinux\nGit\nLarge-volume ETL/data migration\nData validation/reconciliation\nSpark performance tuning\nDistributed-system troubleshooting.\nData Modelling\nStrong experience with:\nlogical and physical data modelling,\nsource-to-target mapping,\nschema/data-type mapping,\nentity relationships,\nkeys and referential relationships,\ndimensional/relational modelling concepts,\nlarge-scale analytical data models.\nTechnical Leadership\nDemonstrated experience:\nleading Data Engineering teams,\ntechnical estimation/planning,\ndetailed solution design,\ncode/design reviews,\nmentoring engineers,\nproduction troubleshooting,\nroot-cause analysis,\ncommunicating technical issues with architects/customer teams.\nPreferred Skills\nApache Polaris/Nessie/Glue.\nOpenShift/Kubernetes/Spark Operator.\nS3-compatible object storage.\nNetApp StorageGRID/ONTAP.\nTaaS/Protegrity or equivalent tokenization technology.\nTrino.\nPostgreSQL.\nControl-M/Airflow.\nCI/CD for Spark/Python applications.\nBanking/financial-services experience.\nPB-scale data migration.\nBronze/Silver/Gold modelling and implementation.\nHiveQL→Spark SQL modernization.","description_format":"text","description_chars":7124,"description_truncated":false,"requirements":{"experience_years_min":10,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Cybersecurity","Health Care","Information Technology","Data Centers & Colocation"],"lifecycle":[{"event":"open","at":"2026-11-03T16:16:36Z"}],"visa":[],"liveness":{"score":99,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.993,"p_room":1,"age_days":1,"expected_fill_days":18,"reasons":["conf:3","wave","velocity","win:early","comp:brand"],"computed_at":"2026-11-05T05:54:30Z"},"pay":null,"html_url":"https://alion.io/job/ntt-data-senior-data-engineer-4","json_url":"https://alion.io/job/ntt-data-senior-data-engineer-4.json","meta":{"generated_at":"2026-11-05T15:13:05Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":19783,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-06T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":50348},"rest":"https://alion.io/mcp/rest/get_company?id=50348"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fntt-data-senior-data-engineer-4"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fntt-data-senior-data-engineer-4"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fntt-data-senior-data-engineer-4"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/ntt-data-senior-data-engineer-4\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fntt-data-senior-data-engineer-4"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3690369}},{"tool":"search_jobs","arguments":{"company":"nttdata.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3690369},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}