Expert Data Engineer

Title: Expert Data Engineer

Shift: 11:00AM - 8:00PM

Work Mode: Hybrid

Location: Bangalore

AI Data Engineering Lead

Introduction to the Job:

The AI Data Engineering Lead owns the design, build, governance, and operational readiness of the pipelines, data products, and retrieval-ready knowledge assets that power AI-enabled products - ensuring AI is built on trusted, permissioned, explainable, and reusable data rather than fragmented or ungoverned sources. The role partners across product, architecture, AI engineering, platform, security, risk, compliance, and MLOps/LLMOps.

Some of the things you’ll be doing

• Own AI-ready data engineering - translate AI product needs into data pipelines and data products for RAG, agents, analytics, and decision support; ensure data is complete, accurate, timely, traceable, and reusable across products and business units; engineer for scale, security, and cost efficiency.

• Build trusted data products - define governed data products across client, entity, product, transaction, finance, risk, vendor, and reference domains, with clear ownership, SLAs, quality expectations, and refresh rules; reduce reliance on spreadsheets and duplicate extracts; partner with data stewards on quality and ownership issues.

• Engineer data for RAG and knowledge-based AI - prepare policies, contracts, regulatory content, and operational knowledge for retrieval; define ingestion, chunking, embedding, indexing, and retirement processes; design vector stores, ranking, and grounding/citation patterns with AI engineering; ensure sources are approved, current, and access-controlled.

• Own data quality and trust controls - define and monitor quality rules (completeness, accuracy, validity, timeliness) for critical data elements; build automated checks and remediation workflows into pipelines; ensure AI products can detect missing, stale, or conflicting data; provide quality evidence for risk and product acceptance.

• Manage metadata, catalog, and lineage - capture business and technical metadata for datasets, pipelines, and vector indexes; ensure assets are cataloged and discoverable; document lineage from source through transformation, embedding, and output for auditability; partner with governance on definitions and ownership.

• Embed access, privacy, and security controls - implement role-, attribute-, jurisdiction-, and purpose-based access so AI products only use authorized data; prevent sensitive data exposure through prompts, logs, embeddings, or outputs; meet minimization, retention, masking, and audit-logging requirements; support security/privacy reviews.

• Support AI data lifecycle operations - monitor pipeline reliability, latency, freshness, and cost; refresh and version datasets, embeddings, and indexes on appropriate schedules; support rollback; partner with MLOps/LLMOps on evaluation and grounding datasets; drive continuous improvement from feedback and production monitoring.

What technical skills, experience, and qualifications do you need?

Data engineering: pipeline engineering and orchestration (ETL/ELT, API/event-driven integration); data product development and modeling; data quality rule design and monitoring; metadata, cataloging, and lineage; master/reference data awareness; access control and sensitive data handling; cloud data platforms and lakehouse/warehouse patterns. Any prior experience with StarBurst will be an added advantage.

AI data: RAG data preparation and document/knowledge ingestion; chunking, embeddings, vector stores, and semantic retrieval; grounding and evaluation datasets; dataset versioning; data prep for AI agents/copilots; freshness, retrieval quality, and source governance; AI data observability; permission-aware retrieval; prevention of sensitive data leakage.

Business and leadership: translating business/product needs into data requirements; strong grasp of data ownership and stewardship; explaining quality and lineage issues to leaders; collaboration across product, AI, architecture, security, and risk; judgment on centralize vs. federate vs. reuse; problem-solving with incomplete or conflicting data; building reusable capabilities over one-off extracts.