Software Engineer II - Data Engineering

Job :

Software Engineer II (Data Engineering)

Location :

Bangalore

Responsibilities :

- Design, develop, and maintain scalable data processing pipelines using the Spark ecosystem, including PySpark, Databricks, and Synapse Notebooks.

- Develop and manage data pipelines using Azure Data Factory, Azure Synapse, and Azure Data Lake to ingest, process, and integrate data from Azure and on-premises sources.

- Implement full and incremental data load strategies, ensuring efficient, reliable, and timely movement of data across source systems and target platforms.

- Perform data cleansing, transformation, aggregation, and modeling to prepare high-quality datasets for analytics, reporting, and downstream consumption.

- Apply Azure security, access control, and data governance practices to ensure secure and compliant data processing and storage.

- Optimize Spark jobs, data pipelines, and SQL workloads for performance, scalability, reliability, and cost efficiency.

- Collaborate with BI and analytics teams to support Power BI reports and dashboards, including datasets/models hosted on Azure Analysis Services (AAS) and Power BI.

- Troubleshoot data pipeline and processing issues, perform root-cause analysis, and implement appropriate fixes to maintain data availability and quality.

- Develop reusable frameworks, processes, and best practices for data ingestion, transformation, validation, and monitoring.

- Work closely with business stakeholders, data analysts, and engineering teams to translate data requirements into scalable and maintainable data solutions.

Key Skills Required :

Data engineering & Analytics :

- 5+ years of experience in statistics and analytics on large-scale datasets

- Strong SQL and data analysis skills, including SQL DW concepts

- Experience with Spark ecosystem: PySpark, Databricks, Synapse Notebooks (1+ year hands-on)

- Proficiency in Azure Data Services:

1. Azure Synapse

2. Azure Data Factory

3. Azure Data Lake

4. Azure security model

- Experience building pipelines for full and incremental data loads from Azure and on-prem sources

- Strong knowledge of data management, processing, cleaning, transformation, aggregation, and modeling

- Working knowledge of Power BI reports and dashboards (AAS / PBI)

Systems, Cloud & DevOps :

- Strong programming skills in Python (Java is a plus)

- Experience with FastAPI, RESTful services, and microservices architecture

- Proficiency with databases such as MySQL, PostgreSQL, and MongoDB

- Experience deploying applications using Docker and Kubernetes

- Experience with streaming and messaging systems: Kafka, Kinesis, RabbitMQ

- Familiarity with Azure Git and DevOps practices

Leadership & Soft Skills :

- Proven experience leading projects end-to-end

- Strong mentoring and team leadership experience

- Excellent written and verbal communication skills

- Ability to work effectively in distributed teams

- Strong understanding of data structures and design patterns

- Exposure to the retail industry is a plus

Good to have skills :

- AI / ML and Agentic AI

Education :

- Bachelor's degree in computer science or a related field (required)

- Master's degree is a plus

Skills

Data Engineering, Azure Data Factory, Spark, Python, SQL, Databricks, Power BI, FastAPI, Kafka, MongoDB, PostgreSQL