Senior Data Engineer – Financial Services

Job Title: Senior Data Engineer - Financial Services (Data Lakehouse & Real-Time Systems)

About the Role

We are seeking seasoned Data Engineering and architecture leaders to architect and deliver a next-generation data lake/house platform for a leading financial services client. This engagement will involve thorough data architecture and engineering components, analysis of complex data and its ecosystem and integrating real-time trade, crypto, securities and fixed income data, analytics, and risk systems for various use-cases. The ideal candidate combines deep domain experience in retail brokerage with hands-on expertise in AWS, MS- Azure Databricks, Spark, and modern data lake/house architectures, and can balance delivery under pressure with architectural foresight and team mentorship. This role requires a deep skill-set in real-time architectural systems and pipelines, adept with financial services specifically, investment domain and a hands-on engineer with a knack of innovative insight on integration of various data architecture/s within the cloud ecosystem.

Key Responsibilities

• Architect and lead the design, implementation, and optimization of large-scale data lake/house systems on Azure Databricks using Spark, Delta Lake, and Python. (advanced)
• Build and optimize streaming pipelines for real-time ingestion, processing, and analytics of trade and market data across asset classes (FX, equities, futures, options, bonds, and crypto). (intermediate)
• Collaborate closely with the CTO and data architects to evolve the end-to-end application architecture for scalability, resilience, and low-latency performance. (advanced)
• Mentor and guide junior and mid-level engineers, establishing best practices in data modelling, orchestration, and operational monitoring.
• Ensure data integrity, lineage, and governance within regulatory and compliance frameworks. (advanced)
• Partner cross-functionally with business and technology teams to align data infrastructure with business KPIs and capital markets product goals.
• Drive automation and CI/CD for data pipelines using Terraform, GitHub Actions, and cloud-native DevOps practices. (advanced)
• Deliver under tight timelines, managing sprint-level execution and ensuring production readiness within a short time-line for GO-LIVE.

Required Skills and Qualifications:

• 15+ years of professional experience, with at least 10 years in financial services and 8+ years in data engineering leadership roles.
• Proven track record building real-time, high-throughput data platforms at enterprise scale.
• Strong proficiency in Azure Databricks, Apache Spark, and Python (PySpark, Panda).
• Solid experience with streaming technologies (Kafka, Kinesis, Event Hubs, or Flink).
• Deep understanding of data lake/house design principles (Cloud Engineering and architecture, AWS, Delta Lake, Iceberg, or Hudi) and data modelling for analytics and ML pipelines.
• Strong background in retail brokerage and trading systems, including exposure to FX, equities, futures, options, bonds, and crypto.
• Experience building or integrating risk management systems, trade surveillance, or crypto exchanges.
• Practical knowledge of data governance, lineage, and metadata management tools (e.g., Azure Purview).
• Proven ability to design scalable, cost-efficient pipelines on Azure using Data Factory, Synapse, and related services.
• Excellent communication, collaboration, and documentation skills, with the ability to influence senior stakeholders.

Preferred Qualifications

• Experience working with microservices-based or event-driven architectures supporting capital markets workloads.
• Familiarity with machine learning pipelines for predictive risk and trading analytics.
• Exposure to Shopify-like multi-tenant or marketplace data architectures.
• Cloud certifications: Microsoft Certified: Azure Data Engineer Associate or equivalent.
• Previous experience in start-up or fast-paced delivery environments with adaptability and velocity.

Core Attributes

• Strong sense of ownership, reliability, and confidentiality in managing critical financial data.
• Ability to balance delivery urgency with architectural integrity.
• A collaborative mentor who uplifts engineering standards across the team.
• Strategic mindset with hands-on execution capability.

Engagement Details

• Number of Positions: 3
• Location: Mumbai
• Engagement Type: Full-Time / Contract-to-Hire
• Timeline: Immediate start; production launch targeted for January
• Reports To: Engagement Lead in conjunction with Client CTO.