Data Engineer

Role Overview

We are seeking a Data Engineer to support a short-term AI application delivery project built on AI platform, which provides capabilities including MLOps, LLM/RAG, conversational UI, low-code data pipelines, and workflow orchestration.

The Data Engineer will focus on secure data integration and data pipeline development. Responsibilities include connecting to client data sources, configuring and implementing pipelines, transforming and preparing data, validating data quality, and documenting the solution for operational handoff. The role will also help define how relational, intermediate workflow, and output data should be stored, structured, and made available to downstream application, analytics, and AI workflows.

The ideal candidate will have strong SQL skills, experience with secure enterprise data connectivity, familiarity with YAML or configuration-driven setup, and hands-on experience building and integrating data pipelines. Relevant data sources may include relational databases, APIs, file-based data, and enterprise systems.

This is an approximately two-month engagement, with the final solution deployed within the client’s on-premises environment.

Key Responsibilities

• Build and configure batch data pipelines using the platform’s low-code data pipeline builder, with support for event- or message-based integration where required.

• Securely connect to client data sources, including relational databases, APIs, file-based sources, on-premises storage, enterprise systems, and Kafka or similar messaging platforms.

• Write and validate SQL for data extraction, transformation, filtering, aggregation, lookup logic, and validation.

• Prepare, transform, clean, and validate data required by AI workflows, ML models, backend services, and frontend applications.

• Define and validate source-to-target mappings, data schemas, field mappings, data types, event/message schemas, and downstream input/output structures.

• Create and maintain YAML or configuration files for pipeline setup, data mappings, data source connections, event/message connections, and workflow inputs.

• Validate data ingestion, including source formats, schemas, load completeness, duplicate and null handling, retry/reprocessing behavior, event/message delivery, data freshness, and downstream availability.

• Perform data quality checks, including completeness, consistency, reconciliation, schema validation, and event/message validation.

• Define how relational, intermediate workflow, event/message, and output data should be stored, structured, and made available to downstream workflows.

• Collaborate with Backend, Frontend, DevOps, and client IT/security teams to ensure correct data formats, secure connectivity, and successful deployment.

• Support testing, troubleshooting, deployment, documentation, and handoff of data pipeline components within the client’s on-premises environment.

Required Skills and Experience

• Strong SQL skills, including joins, aggregations, filtering, CTEs, window functions, and query optimization.

• Experience with data transformation, cleansing, source-to-target mapping, and data validation.

• Hands-on experience building batch data ingestion pipelines from relational databases, APIs, file-based sources, and shared storage, including scheduling, incremental loads, retry handling, error handling, and delivery validation.

• Experience working with structured and semi-structured data, including relational tables, JSON, CSV, Parquet, event messages, and API outputs.

• Familiarity with Kafka or similar messaging technologies, including event/message schemas and basic producer/consumer concepts.

• Familiarity with YAML or configuration-driven data pipeline setup.

• Experience with PostgreSQL, including schema design, indexing, query optimization, and relational data modeling.

• Experience with secure enterprise or on-premises data integration, including network restrictions, credentials, and access controls.

• Strong troubleshooting, testing, documentation, and independent delivery skills.

Preferred Qualifications

• Experience with data governance considerations, including access control, retention, lineage, auditability, and secure handling of client data.

• Familiarity with Python or Java for custom transformation or integration logic where needed.