{"id":1121187,"url":"https://alion.io/job/zenteiq-ai-data-engineer-llm-training-data","title":"AI Data Engineer","company":{"id":1338,"name":"ZenteiQ","domain":"zenteiq.ai","url":"https://alion.io/company/zenteiq-ai","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Keka","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"junior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":14000,"max_usd":32000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":12},"experience_years_min":2,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Beautiful Soup","optional":false},{"name":"Dask","optional":false},{"name":"ETL/ELT","optional":false},{"name":"GCP","optional":false},{"name":"LLM","optional":false},{"name":"Python","optional":false},{"name":"Ray","optional":false},{"name":"Spark","optional":false},{"name":"Hugging Face","optional":true},{"name":"NVIDIA NeMo","optional":true},{"name":"Rust","optional":true},{"name":"Synthetic Data","optional":true}],"status":"live","first_seen_at":"2026-09-04T06:44:58Z","employer_posted_date":"2026-09-04","last_verified_at":"2026-10-17T17:13:09Z","board_verified":true,"closed_at":null,"days_open":43,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":43},"description":"About ZenteiQ\nZenteiQ is a deep-tech company born out of IISc Bangalore, building Scientific Intelligence Infrastructure: physics-native AI for engineering, manufacturing, energy, mobility and national systems. Rather than wrapping general-purpose language models, we train foundation models (BrahmAI) from scratch to reason over thermal, electromagnetic, structural and materials domains, and put them to work through industrial platforms (KogneX) and a talent OS for engineers and researchers (AhamX). We're backed by the IndiaAI Mission (MeitY) and work closely with IISc, ARTPARK and a national AI Hub Network - building the sovereign AI infrastructure that India's engineering and industrial systems will run on.\nAbout the Role\nAt ZenteiQ we're building the data foundation that powers next-generation Large Language Models. As a Data Engineer on the corpus team, you'll build scalable pipelines to source, process, curate, filter and prepare large-scale corpora for training and evaluating foundation models.\nYou'll work closely with AI Researchers, ML Engineers and Platform Engineers to solve challenging data problems across web-scale datasets, unstructured data, multilingual corpora and benchmark datasets.\nWhat You'll Do\nSource and ingest large-scale datasets from Common Crawl, public web sources, open-source repositories and other data sources.\nBuild scalable data processing pipelines for large unstructured corpora.\nDevelop and optimise distributed data pipelines using tools such as Apache Spark, Ray and Dask.\nExtract and process content from web pages and documents using Trafilatura and related web/content extraction libraries.\nDesign data cleaning, filtering, quality assessment and curation pipelines for LLM pretraining corpora.\nDevelop heuristic-based filters to remove low-quality, boilerplate, spam, duplicated or otherwise unsuitable content.\nImplement document-level and corpus-level deduplication strategies.\nIdentify and mitigate benchmark contamination and evaluation-data leakage in training corpora.\nPrepare and curate benchmark and evaluation datasets.\nWork with large-scale datasets stored and processed through GCS/GCP.\nMonitor data quality, pipeline performance, storage and processing efficiency.\nCollaborate with AI Researchers and ML Engineers to continuously improve corpus quality and training-data pipelines.\nWhat We're Looking For\nMust Have\n2-5+ years of experience in Data Engineering, AI Data Engineering, ML Data Engineering or a related field.\nStrong Python programming skills.\nExperience working with large-scale unstructured datasets.\nExperience building data processing / ETL pipelines.\nStrong understanding of data cleaning, filtering, validation and deduplication.\nHands-on experience with exact and near-duplicate detection and deduplication.\nFamiliarity with at least one distributed processing framework: Apache Spark, Ray or Dask.\nUnderstanding of web-scale data ingestion, such as Common Crawl or other large public datasets.\nExperience with HTML parsing and web/document extraction libraries such as Trafilatura, BeautifulSoup, lxml or Resiliparse.\nFamiliarity with GCS/GCP and cloud-based data processing.\nExperience working with large-scale formats such as JSONL, Parquet or Arrow.\nStrong problem-solving and ownership mindset.\nGood to Have\nExperience with LLMs, Foundation Models or Generative AI.\nExperience building or maintaining LLM pretraining corpora.\nFamiliarity with Hugging Face Datasets, DataTrove or NVIDIA NeMo Curator.\nExperience with WARC/WET/WAT and large-scale Common Crawl processing.\nKnowledge of MinHash, fuzzy or semantic deduplication.\nResearch-level understanding of synthetic data generation techniques for model training and evaluation.\nFamiliarity with Rust for high-performance data processing.\nExperience with multilingual or domain-specific corpora.\nWhy ZenteiQ\nBuild real physics-native foundation models from scratch, not another LLM wrapper - deep, defensible technical work.\nBe part of a nationally recognised mission: one of 8 startups selected under the government's IndiaAI Mission to build a sovereign foundation model.\nWork alongside IISc-trained scientists and researchers, in a company founded by an IISc professor.\nSee your work land in real industry pilots across automotive, mobility, defence and industrial R&D, with measurable impact.\nJoin a lean, high-calibre team at an early, high-ownership stage.","description_format":"text","description_chars":4408,"description_truncated":false,"requirements":{"experience_years_min":2,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Machine Learning","STEM Education","Simulation & Digital Twin Software","AI for Science"],"lifecycle":[{"event":"open","at":"2026-09-22T19:09:34Z"}],"visa":[],"liveness":{"score":14,"band":"cold","label":"Long shot","p_open":1,"p_active":0.503,"p_room":0.28,"age_days":42,"expected_fill_days":19,"reasons":["conf:1","win:tail","crowd:junior"],"computed_at":"2026-10-17T05:46:00Z"},"pay":null,"html_url":"https://alion.io/job/zenteiq-ai-data-engineer-llm-training-data","json_url":"https://alion.io/job/zenteiq-ai-data-engineer-llm-training-data.json","meta":{"generated_at":"2026-10-17T18:35:32Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":1407,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1338},"rest":"https://alion.io/mcp/rest/get_company?id=1338"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fzenteiq-ai-data-engineer-llm-training-data"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fzenteiq-ai-data-engineer-llm-training-data"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fzenteiq-ai-data-engineer-llm-training-data"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/zenteiq-ai-data-engineer-llm-training-data\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fzenteiq-ai-data-engineer-llm-training-data"}]}