Consultant - PySpark / Databricks Developer (Intermediate - Senior)

About the Role

We are looking for both permeant and contract for this role.

We're looking for a developer to help build and maintain data pipelines on Databricks for a financial services / wealth management data platform. The platform ingests data from multiple source systems, transforms it through a medallion architecture (bronze → silver → gold) using PySpark, and publishes curated data to downstream consumers via file exports and Kafka. You'll work alongside a senior engineer, writing transform code, fixing pipeline issues, and helping keep the environment healthy across dev/qa/uat.

This is a good fit for someone with solid Python fundamentals and some Spark/SQL exposure who wants to grow into a data engineering specialist. You won't be expected to know Databricks internals on day one - you will be expected to learn fast, ask good questions, and write clean,

What You'll Work On

• Writing and maintaining PySpark transforms (hand-written, not a generic framework) for silver and gold layer tables - things like customer, account, and transaction data models.
• Working with Lakeflow Declarative Pipelines (Databricks' current pipeline framework - successor to DLT) and Auto CDC / SCD Type 2 patterns for change-data-capture merges.
• Querying and validating data in Unity Catalog across environments (dev/qa/uat) using SQL warehouses.
• Debugging failed pipeline runs: reading pipeline event logs, tracing bad records, fixing schema drift or data quality issues.
• Maintaining reference/lookup tables and small utility scripts (PowerShell/Python) used to operate the platform.
• Writing and updating unit/integration tests for transform logic.
• Participating in code review, using Git feature branches and merge requests (GitLab).
• Keeping documentation current when you change how something works.

Must-Have Skills

• Python - comfortable writing clean, readable code; understands functions, modules, basic OOP.
• Databricks
• Some exposure to Apache Spark / PySpark, or strong SQL skills plus a willingness to learn Spark quickly.
• Working knowledge of SQL (joins, aggregations, window functions).
• Basic Git workflow: branches, commits, pull/merge requests.
• Comfortable reading other people's code and stack traces, and debugging methodically (not guess-and-check).

Nice-to-Have Skills

• Direct experience with Databricks (notebooks, jobs, clusters, or SQL warehouses).
• Familiarity with Delta Lake, medallion architecture (bronze/silver/gold), or CDC/SCD concepts.
• Exposure to Azure (this environment runs on Azure Databricks with Azure AD service principal auth).
• Experience with Kafka or other streaming/event systems.
• Experience with CI/CD pipelines (this project uses GitLab CI).
• Prior work in financial services or a regulated data environment.

What Success Looks Like in the First 90 Days

• Can independently pick up a small transform bug or enhancement ticket, make the change, test it, and open a merge request.
• Comfortable running existing operational scripts to check pipeline status, query tables, and diagnose failures without hand-holding.
• Starting to take ownership of the Databricks environments and help write new transforms