[Remote] Senior Databricks Data Architect

Note: The job is a remote job and is open to candidates in USA. AG Coders is seeking a Senior Databricks Data Architect to lead the implementation of scalable, secure, and automated test data management frameworks using Databricks. The role focuses on data governance, secure test data provisioning, and collaboration with QA, application development, data governance, and information security teams.

Responsibilities

• Lead the rollout of Test Data Management (TDM) frameworks across multiple projects on Databricks
• Establish governance controls to protect sensitive data in non-production environments
• Design and develop scalable pipelines for test data generation, masking, and refresh
• Implement metadata management, data lineage, and data governance capabilities
• Define and implement synthetic data strategies for non-production environments
• Enable self-service access to secure and compliant test datasets
• Develop reusable and automated data provisioning and data lifecycle management frameworks
• Collaborate with QA, application development, data governance, and information security teams

Skills

• 8+ years of experience in Data Engineering, Data Architecture, or Data Platform roles
• 3+ years of hands-on experience with Databricks, including Lakehouse, Delta Lake, Unity Catalog, and PySpark
• Strong experience implementing Medallion Architecture (Bronze, Silver, and Gold layers)
• Proven experience with Unity Catalog implementation and data governance rollouts
• Strong understanding of enterprise data governance, including:
• Role-Based Access Control (RBAC)
• Metadata management and data classification
• Data lineage and access controls
• Experience building scalable data ingestion, transformation, and provisioning pipelines
• Hands-on experience with Azure (ADF, Azure DevOps, Synapse) and/or AWS (S3, Lambda, Glue, Redshift, MWAA)
• Strong Python and PySpark development skills
• Test Data Management (TDM), including data masking, de-identification, and synthetic data generation
• AI-powered data discovery and data classification
• Generative AI and Databricks Mosaic AI
• Data observability and monitoring frameworks
• Experience driving cross-team framework rollouts and technical enablement
• CI/CD and Infrastructure as Code (IaC), including Terraform, Azure DevOps, and GitHub Actions
• Knowledge of regulatory and data privacy requirements such as PII, GDPR, CCPA, and HIPAA

Benefits

• Remote (in US)

Company Overview