Senior Apache Spark Technical Specialist - Scala, Python

Job Summary

We are seeking a Senior Solution Architect with deep experience designing and delivering enterprise-scale big data, streaming, and real-time data integration solutions.

The role is focused on solution architecture across Cloudera-based big data platforms, Kafka-based streaming applications, and real-time data integration / analytics solutions, including cloud-hosted data platforms such as Cloudera on AWS or equivalent AWS-native services.

The role also provides architectural leadership across Data Solutions, helping shape architecture strategy, design standards, engineering best practices, and the future evolution of the Bank's data platform capabilities.

Key Responsibilities

• Own the end-to-end solution architecture for large-scale distributed data, streaming, and data integration solutions.
• Act as the lead architect for Data Solutions distributed data, integration and streaming platforms, providing architectural direction, governance, design assurance and technical leadership across initiatives, while influencing the evolution of the target architecture and ensuring alignment with enterprise technology strategy, engineering standards and long-term platform objectives.
• Accountable for solution architecture across the Enterprise Data Lake, Operational Datastore and associated data platform services within the Cloudera CDP environment on AWS.
• Design and document scalable distributed data and streaming solutions across Cloudera, Kafka, API, and cloud platforms, including ingestion, processing, storage, and consumption patterns.
• Provide architectural oversight across solution delivery, ensuring consistency, quality and alignment with approved designs.
• Work with engineering teams to ensure solutions are deliverable, scalable, secure, supportable, and aligned to enterprise architecture principles.
• Review technical designs produced by engineering teams and ensure alignment with the approved solution architecture.
• Identify technical risks, dependencies, assumptions, non-functional requirements, and design trade-offs.
• Support delivery planning by helping break solutions into iterative delivery increments and sprint-level outcomes.
• Represent solution designs at architecture review boards and governance forums, engaging effectively with business stakeholders, delivery teams, and enterprise architects.
• Develop and promote architectural patterns, engineering standards and best practices across the data platform ecosystem.

Skill Requirements

The successful candidate must demonstrate strong practical experience across many of the following areas:

• Distributed data platforms and data lake architecture on Cloudera CDH/CDP or equivalent platforms.
• Event streaming and real-time integration technologies, including Kafka and related messaging patterns.
• Large-scale data processing and transformation frameworks such as Spark, Flink, Hive, Impala or equivalent technologies.
• API and service-based integration architectures using technologies such as Java, Spring Boot, and REST services.
• NoSQL and distributed data storage technologies including HBase, Cassandra, or equivalent platforms.
• Cloud-hosted data platforms and services, preferably on AWS.
• Modern engineering practices including DevOps, CI/CD, observability, resilience, and operational support.

Other Requirements

Desirable Experience

• Financial services or banking experience.
• Experience with regulated, secure, enterprise-scale data platforms.
• Experience presenting to architecture governance boards.
• Experience leading architecture communities, defining standards, or contributing to capability development within an architecture practice.
• Exposure to enterprise data governance, lineage, metadata, and data quality tooling.
• Experience with cloud-native data services on AWS.

Candidate Profile

This role is suited to a hands-on Senior Solution Architect who can design and guide the delivery of complex data and streaming solutions.

The ideal candidate will:

• Be an architect first, with strong hands-on engineering awareness and technical credibility.
• Have designed and delivered production-scale distributed data, streaming, and integration solutions.
• Understand both architecture-level trade-offs and engineering-level implementation constraints.
• Be comfortable challenging, reviewing, and improving technical designs.
• Be able to influence senior stakeholders, lead architectural decision-making, and engage deeply with engineering teams.
• Have practical experience delivering scalable, resilient, secure, real-time or near-real-time data platforms and promoting technical excellence across teams.

Important Clarification

This is not a role for candidates whose experience is primarily in:

• Data analysis
• Data modelling only
• BI reporting incl Power BI / Tableau reporting
• Dashboard development
• SQL reporting
• Data governance only
• Business analysis
• Conceptual data architecture without implementation experience

Candidates must have demonstrable experience in solution architecture for distributed big data platforms, streaming systems, event-driven integration, or real-time data applications.