Senior Data Engineer

Responsibilities:

• Design and develop advanced data engineering solutions that enable data pipelines, visualisation, and analytical tools.
• Architect and implement scalable data platforms across on-premises, cloud, and hybrid environments.
• Build, optimise, and operate complex batch and real-time and near-real-time streaming data pipelines.
• Build, optimise, and operate low-latency, sub-second APIs on highly scalable, fault-tolerant architectures.
• Design and maintain high-throughput data pipelines to support agentic AI workloads and real-time multi-agent MCP orchestration.
• Build and maintain enterprise-scale Graph Databases that power organisational knowledge graphs and connected data experiences.
• Perform data wrangling, exploration, and discovery across heterogeneous data sources to generate insights.
• Lead development of large-scale data processing solutions using Databricks and Spark technologies.
• Design and optimise relational, NoSQL, and analytical data models.
• Contribute to team knowledge sharing and advancement of data engineering standards and capabilities.
• Mentor engineers to strengthen technical skills, delivery quality, and professional growth.
• Support project definition activities including estimation, planning, and scoping in partnership with management.

Requirements:

• Bachelor's Degree in Computer Science, Computer Engineering, or a related field, or equivalent practical experience.
• 5-8 years of hands-on experience designing, building, and supporting data engineering solutions.
• Strong experience developing and migrating data solutions across cloud platforms.
• Demonstrated technical leadership and mentoring experience.
• Strong analytical and problem-solving skills applied to complex data challenges.
• Ability to manage multiple concurrent initiatives with strong organisational and prioritisation skills.
• Passion for learning and applying new data and platform technologies.

Core Data Engineering and Platforms:

• Advanced experience designing and building complex data pipelines using Python and SQL.
• Strong experience with cloud platforms and services, including: Azure (Data Factory, Data Lake, Event Hub, Functions, Web Apps, Cosmos DB) or GCP (Google Spanner, BigQuery).
• Databricks with PySpark/ Spark SQL/ Scala Spark, including cluster configuration, autoscaling, and performance optimisation (e. g., Photon).
• Snowflake data warehousing, including schema design (star/snowflake), query optimisation, and cost/performance tuning.
• Expertise in SQL, NoSQL, and relational database design and development.

APIs, Streaming and Integration:

• Proficiency in designing and consuming RESTful APIs with secure authentication and authorisation (OAuth 2.0 JWT, API keys).
• Experience using Postman and Swagger/OpenAPI for API testing, documentation, and validation.
• Working knowledge of message queuing, stream processing, and highly scalable big-data data stores.

Engineering Practices:

• Advanced knowledge of data pipeline development using Python and experience with languages such as SQL, DAX, Java, Scala, and/or Go.
• Experience performing root-cause analysis and using technology to solve complex business problems.
• Experience adopting AI-assisted development tools and building AI-powered agents or agentic workflows to automate engineering tasks, accelerate delivery, and enhance data platform capabilities.
• Experience with NL-to-SQL query generation using Databricks Genie and Snowflake Cortex to enable natural language access to data assets.
• Hands-on experience with Unity Catalogue for data governance, fine-grained access control, and end-to-end data lineage.
• Experience developing AI-assisted dbt models, including AI-generated transformation logic, automated documentation, and intelligent testing.
• Familiarity with AI-powered data quality tools for automated profiling, anomaly detection, and pipeline observability.

Nice to Have:

• Experience with Iceberg table design and optimisation.
• Experience with Unity Catalogue governance and security.
• Cloud platform certifications such as Azure Certified Solutions Architect, Azure Cloud Practitioner, or MCSA.
• Experience contributing to enterprise data governance, security, or compliance initiatives.
• Experience with ML feature platforms (e. g., Feast, Tecton, or Databricks Feature Store) and vector database engineering (e. g., Pinecone, Weaviate, pgvector).
• Experience designing and operating streaming AI pipelines and applying MLOps practices to data engineering workflows.
• Exposure to end-to-end ML pipeline ownership and real-time AI inference plumbing in production environments.

Skills

azure, bigquery, data engineer, data engineering, databricks, gcp, pyspark, python, snowflake, spanner, spark, sql