Data Engineer / AI Engineer

Job Description – Senior Data Engineer / AI Engineer

Experience: 8+ Years

Locations: Bengaluru & Pune

Work Mode: Hybrid (Work From Office)

Role Type: Full-Time

Job Summary

We are looking for an experienced Senior Data Engineer / AI Engineer with strong expertise in Big Data technologies, data engineering, distributed computing, and AI/ML engineering.

The ideal candidate should have hands-on experience designing and developing scalable data pipelines, processing large volumes of structured and unstructured data, and building AI-powered data solutions. The candidate should possess strong programming skills, experience with modern Big Data ecosystems, and the ability to integrate AI/ML capabilities into enterprise data platforms.

Key Roles & Responsibilities

1. Data Engineering & Big Data Development

• Design, develop, and maintain scalable and high-performance data pipelines.
• Build robust ETL/ELT workflows for processing structured, semi-structured, and unstructured data.
• Work with large-scale distributed data processing frameworks.
• Develop data ingestion, transformation, validation, and orchestration workflows.
• Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
• Implement batch and real-time data processing solutions.

2. Big Data Technologies

• Work extensively with Apache Spark, PySpark, Hadoop, Hive, and Kafka.
• Develop distributed data processing applications using Spark and PySpark.
• Implement real-time streaming solutions using Kafka or similar technologies.
• Work with data lakes, data warehouses, and lakehouse architectures.
• Optimize SQL queries, data models, and large-scale data processing jobs.
• Experience with Delta Lake, Apache Iceberg, or Hudi is an added advantage.

3. AI / Machine Learning Engineering

• Design and develop AI/ML solutions integrated with enterprise data platforms.
• Build and deploy machine learning models and data-driven applications.
• Work with Python-based AI/ML frameworks and libraries.
• Develop AI-powered workflows using LLMs, Generative AI, and related technologies.
• Integrate LLM APIs and AI capabilities into existing data pipelines and enterprise applications.
• Work on RAG-based solutions, embeddings, vector databases, and semantic search.
• Implement model evaluation, monitoring, and performance optimization.

4. Cloud & Data Platform Technologies

• Hands-on experience with cloud-based data platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP).
• Experience with services such as AWS S3, Glue, EMR, Redshift, Azure Data Factory, Azure Databricks, or Google BigQuery.
• Build and manage scalable cloud-native data engineering solutions.
• Implement data storage, processing, and analytics solutions across cloud environments.

5. Data Architecture & Governance

• Design scalable data architectures, data models, and data integration frameworks.
• Implement data quality checks, data validation, and data governance practices.
• Ensure data security, privacy, and compliance across data platforms.
• Work closely with data scientists, AI engineers, architects, and business stakeholders.

Required Technical Skills

Skill CategoryRequired Skills

Programming

Python, PySpark, SQL

Big Data

Apache Spark, Hadoop, Hive

Streaming

Apache Kafka, Spark Streaming

Data Engineering

ETL/ELT, Data Pipelines, Data Warehousing

Data Architecture

Data Lake, Lakehouse, Delta Lake

AI/ML

Machine Learning, Generative AI, LLMs

AI Frameworks

LangChain, LlamaIndex, Scikit-learn (preferred)

Vector Databases

Pinecone, FAISS, Chroma, Milvus

Cloud

AWS / Azure / GCP

Orchestration

Apache Airflow, Azure Data Factory

Databases

SQL, NoSQL, PostgreSQL, MongoDB

Mandatory Requirements

• 8+ years of overall experience in Data Engineering, Big Data Engineering, or AI Engineering.
• Strong hands-on experience with Python, PySpark, and SQL.
• Extensive experience working with Big Data technologies such as Spark, Hadoop, Kafka, and Hive.
• Strong understanding of distributed computing and large-scale data processing.
• Experience developing and optimizing complex ETL/ELT data pipelines.
• Good understanding of AI/ML concepts and their integration with data engineering solutions.
• Experience working with cloud-based data platforms.
• Strong analytical, problem-solving, and communication skills.