Forward Deployed Engineer - Data Engineering

Databricks Forward Deployed Engineer :

Role Overview :

The Databricks Forward Deployed Engineer (FDE) is a senior client-facing engineer embedded with enterprise clients to design, build, and optimise production-grade data platforms on Databricks. The role requires deep Spark and Databricks technical expertise combined with strong client communication skills.

Key Responsibilities :

Client Engagement & Solution Design :

- Lead technical discovery sessions to understand client data platform challenges and objectives

- Design end-to-end Lakehouse architectures (Medallion, streaming, batch, and hybrid patterns)

- Present architecture recommendations and trade-off analyses to technical and executive stakeholders

- Produce solution architecture documents, data flow diagrams, and design decision records

Platform Engineering :

- Build production data pipelines using PySpark, Spark SQL, and Delta Lake on Databricks

- Implement ingestion patterns using AutoLoader, Kafka, and event-driven architectures

- Implement SCD Type 1/2/3 and CDC patterns for dimension and fact table management

- Optimise Delta Lake tables (OPTIMIZE, VACUUM, Liquid Clustering, Z-Ordering, Deletion Vectors)

- Right-size and configure Databricks clusters

- Configure Unity Catalog (data lineage, access controls, workspace federation)

Performance & Production Engineering :

- Diagnose performance bottlenecks using Spark UI

- Implement CI/CD pipelines using Databricks Asset Bundles (DABs), GitHub Actions, or Azure DevOps

- Define data quality checks, SLA monitoring, and production alerting frameworks

GenAI & Advanced Analytics :

- Support GenAI solution design (RAG pipelines, vector store integration, LLM evaluation)

- Leverage Mosaic AI and Databricks Genie for ML and analytics workloads

Required Qualifications :

- 5+ years of data engineering experience in production environments

- 2+ years of hands-on Databricks or Apache Spark platform delivery at scale

- Prior client-facing, consulting, or embedded engineering experience

Technical Skills :

- Deep PySpark and Spark SQL proficiency

- Hands-on Delta Lake expertise

- Strong Databricks cluster architecture knowledge

- Strong SQL (window functions, CTEs, analytical queries)

- CI/CD hands-on (DABs, notebooks-as-code)

- Unity Catalog or equivalent data governance frameworks

Communication & Soft Skills :

- Ability to present complex technical decisions clearly to non-technical stakeholders

- Structured problem-solving and ownership mindset

Preferred Qualifications :

- Databricks Certified Data Engineer Associate or Professional certification

- Experience with Structured Streaming, Delta Live Tables, or Kafka

- GenAI / LLM exposure (RAG, vector databases, hallucination mitigation)

- Cloud platform experience (Azure preferred, AWS, or GCP)

- Prior experience within a DPP, GSI, or consulting delivery model

Skills

Python, Azure, Generative AI, Databricks, PySpark, Delta Lake, SQL, Kafka, Forward Deployment Engineering