Specialist Software Engineer - Java + React

Responsibilities

MSA Role: Software Engineer / Senior Software Engineer – Databricks & AWS

Experience: 5–8 Years

• Design, develop, and maintain scalable data engineering solutions using Databricks, PySpark, SQL, and Delta Lake.
• Build and optimize batch and real-time data pipelines on AWS using services such as S3, Lambda, Glue, EMR, Redshift, and related cloud-native components.
• Develop, implement, and support Delta Live Tables (DLT) pipelines for reliable, high-quality, and auditable data processing.
• Work with Unity Catalog to implement data governance, access control, lineage, and security best practices.
• Develop and optimize ETL/ELT pipelines with a focus on performance tuning, scalability, reliability, and cost efficiency.
• Implement infrastructure and platform automation using Terraform and Infrastructure as Code (IaC) principles.
• Build and maintain CI/CD pipelines using Jenkins, Git, and related DevOps tools to enable automated deployments and testing.
• Support deployment, monitoring, and maintenance of Databricks workloads across development, testing, and production environments.
• Collaborate with architects, product owners, data scientists, and business stakeholders to understand requirements and deliver high-quality solutions.
• Participate in cloud migration, modernization, and data platform transformation initiatives on AWS.
• Contribute to data quality, observability, logging, monitoring, and alerting frameworks to ensure platform reliability.
• Work with APIs, data integration frameworks, and external systems to enable seamless data exchange across platforms.
• Troubleshoot production issues, perform root cause analysis, and drive continuous improvement initiatives.
• Follow software engineering best practices, participate in code reviews, and ensure adherence to coding, security, and operational standards.
• Work effectively within Agile teams, contributing to sprint planning, estimation, development, and release activities.
• Collaborate with global teams and support business-critical deliveries across different time zones when required.

Profile required

Profile Required:
Senior‑level Software Engineer with deep Databricks, AWS, and Python data platform expertise, capable of designing and leading enterprise‑scale AI‑enabled data solutions.

Minimum Qualifications:

• Deep hands‑on experience with Databricks (PySpark, SQL, Delta Lake) in large‑scale production environments.
• Proven experience designing and leading end‑to‑end data pipelines on Databricks.
• Strong hands‑on experience with Delta Live Tables (DLT) for governed data processing.
• Advanced experience with Unity Catalog for data governance, access control, lineage, and metadata.
• Expertise in Spark performance tuning, scalability, and cost optimization.
• Experience managing Databricks workflows, monitoring, and production operations.
• Strong knowledge of cloud‑based data platforms (AWS preferred).
• Proven ability to mentor engineers and lead technical delivery.

Preferred Qualifications:

• Experience building and operating AI agents for workflow automation and decision support.
• Strong understanding of data observability, platform reliability, and FinOps practices in Databricks.
• Experience working in regulated or financial services domains.
• Exposure to enterprise‑scale AI governance, compliance, and audit requirements.
• Ability to influence architecture decisions and align stakeholders.

• Excellent communication skills with both technical and non‑technical audiences.