Senior Azure Databricks Architect

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Azure Databricks Architect based in Canada.

This is a senior, hands-on architecture role focused on designing and deploying secure, enterprise-grade Azure Databricks environments.

You will lead complex cloud data platform initiatives, working closely with security, networking, cloud platform, and data engineering teams.

The role combines architecture leadership with practical implementation, requiring you to make technical decisions, write and review Terraform, and troubleshoot challenging connectivity issues.

You will design secure networking patterns involving Azure Private Link, private DNS, VNet injection, Secure Cluster Connectivity, and no-public-IP configurations.

Your work will help enterprise clients establish scalable, governed, and operationally robust data platforms across development, test, and production environments.

You will also provide technical direction to engineers and client stakeholders while establishing repeatable infrastructure-as-code and CI/CD practices.

This opportunity is ideal for an experienced Azure and Databricks specialist who enjoys solving complex infrastructure challenges and taking solutions from architecture through production deployment

Accountabilities:

• Design secure, scalable, and highly available Azure Databricks architectures aligned with enterprise security, connectivity, governance, and operational requirements.
• Architect and deploy Azure Databricks workspaces using Terraform and infrastructure-as-code practices, with direct ownership of implementation rather than simply operating within preconfigured environments.
• Design and implement Azure Private Link and private endpoint connectivity for Databricks, including front-end, back-end, and browser authentication connectivity where required.
• Design customer-managed VNets and VNet-injected Databricks deployments, including Secure Cluster Connectivity and no-public-IP configurations.
• Architect private DNS zones, DNS resolution, routing, network security groups, user-defined routes, and firewall configurations.
• Integrate Databricks environments into hub-and-spoke and other enterprise Azure network architectures.
• Troubleshoot complex DNS, routing, authentication, firewall, and network connectivity issues in collaboration with client security and networking teams.
• Design identity and access management solutions using Microsoft Entra ID, Databricks Unity Catalog, service principals, and related security controls.
• Establish standardized architecture patterns and deployment practices across development, testing, and production environments.
• Implement CI/CD pipelines and infrastructure-as-code processes for Databricks infrastructure and platform configuration.
• Design secure connectivity between Databricks and Azure services such as ADLS Gen2, Key Vault, Azure SQL, Event Hubs, and other enterprise data services.
• Produce architecture diagrams, technical design decisions, deployment documentation, and operational runbooks.
• Provide technical leadership and guidance to data engineers, cloud engineers, security teams, networking teams, and client stakeholders.
• Translate business and security requirements into practical, scalable architecture decisions and guide solutions through implementation and production deployment.

Requirements:

• Significant hands-on experience architecting and implementing Azure Databricks environments in enterprise settings.
• Demonstrated experience personally deploying Azure Databricks using Terraform, including writing, reviewing, and troubleshooting infrastructure-as-code.
• Proven hands-on experience implementing Azure Private Link and private endpoints for Databricks.
• Strong experience with VNet injection, customer-managed VNets, Secure Cluster Connectivity, and no-public-IP Databricks deployments.
• Deep understanding of Azure networking, including private DNS zones, DNS resolution, routing, network security groups, user-defined routes, and Azure firewalls.
• Experience designing and implementing hub-and-spoke Azure network architectures.
• Experience with ExpressRoute and/or VPN connectivity in enterprise environments.
• Strong knowledge of Terraform resources and infrastructure-as-code deployment patterns.
• Experience securing connectivity between Databricks and services such as ADLS Gen2, Key Vault, Azure SQL, Event Hubs, or other Azure data services.
• Strong knowledge of Databricks security, identity and access management, Unity Catalog, Microsoft Entra ID, and service principals.
• Experience establishing CI/CD and infrastructure-as-code practices for enterprise data platforms.
• Ability to diagnose and resolve complex issues involving front-end access, back-end connectivity, browser authentication, private DNS, routing, and cluster connectivity.
• Ability to translate security, technical, and business requirements into clear architecture decisions and practical implementation plans.
• Strong written and verbal communication skills, with the ability to work directly with clients and diverse technical stakeholders.
• Ability to combine strong architectural judgment with a hands-on approach and guide solutions from design through implementation.
• Databricks architecture or engineering certification is preferred.
• Microsoft Azure certification is preferred.
• Experience with Databricks Asset Bundles, Azure DevOps, or GitHub Actions is an advantage.
• Knowledge of data governance, medallion architecture, observability, and cloud cost management is desirable.
• Consulting or professional services experience is a plus.
• Candidates must be based in Canada; the position is fully remote within Canada.

Benefits:

• Remote position within Canada, offering flexibility to work from home.
• Opportunity to work on complex, enterprise-scale Azure, Databricks, and cloud data platform initiatives.
• Hands-on exposure to advanced cloud networking, infrastructure-as-code, data security, governance, and AI/data technologies.
• Opportunity to work directly with enterprise clients and influence critical architecture and technology decisions.
• Collaborative environment involving cloud, data engineering, security, networking, and technology professionals.
• Contract engagement within the technology function.
• Opportunity to take ownership of solutions from initial architecture and design through implementation and production deployment.
• Professional growth through exposure to diverse enterprise cloud and data modernization challenges.