Databricks Administrator

Who You Are

We are seeking an experienced Azure Databricks Administrator to manage, maintain, secure, and optimize Databricks environments in Azure. This role is focused on platform administration, governance, operational support, performance, reliability, and cost efficiency across the organization’s data analytics infrastructure.

The successful candidate will serve as a platform administrator and operational partner to data engineering, analytics, cybersecurity, compliance, and business teams, helping ensure Databricks is reliable, well-governed, secure, and effectively supported.

What You'll Do

• Configure, maintain, and administer Azure Databricks workspaces and clusters across development, staging, and production environments.
• Monitor cluster health, performance metrics, resource utilization, and platform availability.
• Optimize cluster, SQL warehouse, and workspace configurations for performance and cost efficiency.
• Manage workspace governance, including job scheduling, notebook organization, audit logging, and operational standards.
• Administer user access, workspace permissions, SQL warehouse configurations, and group-based security controls.
• Troubleshoot infrastructure issues, performance bottlenecks, connectivity problems, and workload failures.
• Support disaster recovery and high-availability planning, validation, and operational readiness for Databricks workloads.
• Collaborate with data engineering teams to establish Databricks usage standards, support workload reliability, troubleshoot platform issues, optimize Delta Lake performance, and promote governed platform adoption.
• Administer and enforce security policies, RBAC, encryption standards, and access controls.
• Maintain documentation and runbooks for operational procedures.
• Support capacity planning, cost optimization, and platform improvement initiatives.

Security Governance

• Maintain security governance standards for Azure Databricks in a financial services environment, including RBAC, least-privilege access, privileged access controls, and group-based permissions.
• Administer Unity Catalog permissions, metastore access, catalogs, schemas, tables, volumes, and external locations to enforce governed data access.
• Manage workspace permissions, admin roles, service principals, groups, SCIM provisioning, and Microsoft Entra ID integration to support secure identity governance.
• Configure and review compute policies, cluster permissions, instance pool access, SQL warehouse permissions, and job ownership to reduce unnecessary privilege and control platform usage.
• Oversee secrets management, token usage, storage credential access, and Key Vault-backed configurations to protect sensitive financial, customer, and operational data.
• Coordinate recurring access reviews, access certification, and timely remediation of excessive, inappropriate, or outdated permissions to support internal control requirements.
• Review Databricks audit logs, security events, access changes, and control findings to identify risks, policy exceptions, segregation-of-duties concerns, and required remediation actions.
• Support audit and regulatory readiness by maintaining evidence for access controls, configuration standards, policy enforcement, operational procedures, and change management activities.
• Partner with cybersecurity, compliance, risk, infrastructure, and data governance teams to align Databricks controls with financial services security, privacy, and compliance expectations.

What You'll Need

• 5+ years of experience administering cloud-based data platforms or analytics environments, with a focus on operational support, governance, and reliability.
• 3+ years of hands-on experience administering Azure Databricks workspaces, Unity Catalog, clusters, jobs, SQL warehouses, and platform settings.
• Strong experience managing Databricks users, groups, workspace permissions, admin roles, service principals, SCIM provisioning, and Microsoft Entra ID integration.
• Experience administering Unity Catalog, including metastores, catalogs, schemas, tables, volumes, permissions, storage credentials, and external locations.
• Ability to configure and maintain compute policies, cluster permissions, instance pools, SQL warehouses, job ownership, and platform usage controls.
• Experience monitoring Databricks platform health, cluster performance, job execution, workload failures, audit logs, and resource utilization.
• Strong troubleshooting skills for Databricks platform issues, Spark workloads, connectivity problems, permission errors, and performance bottlenecks.
• Working knowledge of Python, SQL, PowerShell, Databricks CLI, REST APIs, and automation practices used to support platform administration.
• Experience supporting operational controls, access reviews, audit evidence, change management, runbooks, and platform documentation.
• Knowledge of Databricks security governance, RBAC, least privilege, encryption, secrets management, audit readiness, and secure data access patterns.
• Understanding of Azure networking and security concepts, including private endpoints, firewalls, network security groups, storage access, Key Vault, and secure connectivity.
• Experience partnering with data engineering teams to support reliable pipelines, workload troubleshooting, performance tuning, platform standards, and secure Databricks operations.
• Travel requirement: All roles require travel during the first 6 months of employment to attend the BHG Oars in Experience. After the initial 6 months, ongoing travel is expected at approximately ~5% annually for periodic team on-sites, team meetings, and cross-functional collaboration.
• Work Authorization: This role is not eligible for employer-provided immigration sponsorship. To be considered, applicants must already possess valid U.S. work authorization that will not require sponsorship now or in the future.

Technical Skills & Technologies

• Databricks Platform Administration: Workspaces, workspace settings, admin console, clusters, jobs, notebooks, SQL warehouses, instance pools, compute policies, and audit logs.
• Unity Catalog Administration: Metastores, catalogs, schemas, tables, volumes, permissions, storage credentials, external locations, and governed data access.
• Identity and Access Management: Users, groups, service principals, workspace permissions, admin roles, RBAC, SCIM provisioning, Microsoft Entra ID integration, and least-privilege access controls.
• Security and Governance: Secrets management, token usage, encryption, Key Vault-backed configurations, access reviews, audit evidence, policy enforcement, and regulatory readiness.
• Monitoring and Troubleshooting: Cluster health, job failures, SQL warehouse performance, Spark workload issues, audit log review, resource utilization, connectivity issues, and performance bottlenecks.
• Automation and Administration Tools: Databricks CLI, REST APIs, Terraform awareness, Python, PowerShell, SQL, runbooks, operational scripts, and platform documentation.
• Azure Platform Integration: Azure Storage, Key Vault, private endpoints, virtual networks, firewalls, network security groups, Azure Monitor, Log Analytics, and secure connectivity patterns.

Preferred Qualifications

• Databricks Platform Administrator certification or equivalent hands-on Azure Databricks administration experience.
• Experience administering Unity Catalog, workspace security, compute policies, SQL warehouses, service principals, audit logs, and platform governance controls.
• Experience supporting Databricks in financial services or another regulated environment with strong audit, access review, and compliance requirements.
• Experience creating runbooks, operational standards, monitoring procedures, access review processes, and administrator documentation for Databricks environments.
• Background in database administration, cloud platform administration, or data platform operations supporting enterprise analytics workloads.