Senior Databricks Engineer (Remote)

Role Overview

We are looking for a Senior Databricks Engineer with strong experience in Azure data engineering, Databricks and data lake technologies, along with an understanding and working experience of IT Security/Cybersecurity use cases. The ideal candidate is a generalist who can work across security data engineering, analytics, Data Science and emerging GenAI use cases.

Key Responsibilities

• Design, develop and maintain scalable data pipelines for IT Security and Cybersecurity use cases.
• Build and manage data solutions using Azure Data Lake Storage (ADLS/ADLS Gen2), Azure Databricks and Azure data services.
• Ingest, transform and process security data from multiple sources such as SIEM, endpoint, identity, network, cloud and application security platforms.
• Develop Databricks/PySpark solutions for processing and analysing large volumes of security telemetry.
• Work with security logs, alerts, vulnerability data, threat intelligence, authentication data and other security datasets.
• Work with Azure Security services and understand security-related data generated across the Azure/Microsoft security ecosystem.
• Develop data models and curated datasets to support Security Operations, Threat Detection, Risk Analytics and reporting.
• Collaborate with Data Scientists to prepare datasets for machine learning, anomaly detection, behavioural analytics and predictive use cases.
• Support GenAI/LLM/RAG use cases involving security data, knowledge retrieval and intelligent security analytics.
• Develop automation scripts using Python Build and maintain Power BI dashboards and reports for security metrics, trends, risks and operational KPIs.
• Implement data quality, governance, access controls and monitoring across data pipelines. Troubleshoot data ingestion, transformation and pipeline issues and optimize performance and cost.
• Collaborate with Cybersecurity, SOC, Data Engineering, Data Science and Cloud Engineering teams.

Required Skills & Experience

• Senior Databricks Engineer with proven experience delivering complex data and analytics solutions in enterprise environments.
• Strong hands-on experience with Data Science, LLMs, Agentic AI frameworks, MCP, and RAG architectures, including designing, building, implementing, and troubleshooting solutions in production.
• Minimum 10 years of hands-on Databricks experience, including architecture, development, optimization, operations, and support of large-scale data platforms.
• Demonstrated experience working with cybersecurity data and analytics, translating security requirements into actionable insights, detection capabilities, and reporting.
• Hands-on experience with Microsoft security platforms, particularly Microsoft Sentinel, Microsoft Defender, and Azure Data Explorer (ADX), including data ingestion, analytics, investigations, and operational use cases.
• Proven ability to lead initiatives end-to-end, from requirements gathering and solution design through implementation, deployment, and operationalization.
• Ability to mentor and guide junior engineers, providing technical leadership, coaching, and best-practice recommendations.
• Responsible for performing quality assurance and technical reviews of own work and the work of junior Data Engineers/Data Scientists, ensuring coding standards, solution quality, documentation, and operational readiness.
• Ability to provide concrete examples of delivered solutions, technical challenges solved, architecture decisions made, and measurable business or security outcomes.
• Strong ownership mindset with a track record of personally driving implementation, not only coordinating or overseeing delivery.

Data Science / AI Exposure

• Good understanding of Data Science concepts, including statistics, machine learning, anomaly detection and feature engineering.
• Ability to work with Data Scientists and understand ML/data requirements. Exposure to GenAI, LLM, RAG or AI-powered analytics is preferred.
• Ability to explore and analyse large security datasets and identify meaningful patterns and trends.

Preferred Certifications:

• Databricks Certified Data Engineer Associate / Professional or equivalent Databricks certification.
• Microsoft security certifications such as SC-200 – Security Operations Analyst Associate, SC-500 – Cloud and AI Security Engineer Associate
• Other relevant certifications such as Security+, CEH, CISSP or equivalent are an advantage.