Principal Data Engineer

About BayRock Labs

At BayRock Labs, we pioneer innovative tech solutions that drive business transformation. As a leading product engineering firm based in Silicon Valley, we provide full-cycle product development, leveraging cutting-edge technologies in AI, ML, and data analytics. Our collaborative, inclusive culture fosters professional growth and work-life balance. Join us to work on ground-breaking projects and be part of a team that values excellence, integrity, and innovation. Together, let's redefine what's possible in technology.

We are looking for a highly skilled Senior Data Engineer / Data Architect with deep hands-on expertise in Azure, Databricks, PySpark, Python, SQL, and modern data warehousing. This role is ideal for someone who has built large-scale data platforms, designed enterprise-grade architectures, and delivered high-quality data products in cloud environments.

Core Technical Expertise

Azure Cloud Platform

• ADLS Gen2, ADF, Synapse, Azure SQL, Key Vault, App Services
• Cloud-native architecture design, scalability, and cost optimization
• Security & governance: RBAC, Managed Identity, Private Endpoints

Databricks Engineering

• End-to-end development on Azure Databricks
• Lakehouse architecture implementation
• Delta Live Tables (DLT), Unity Catalog
• Performance tuning, cluster optimization
• CI/CD for Databricks workloads

Big Data & Processing Frameworks

• Apache Spark with advanced PySpark transformations
• Structured Streaming & batch data pipelines
• Delta Lake architecture & optimization

Programming & Querying

• Python for automation, data engineering, and orchestration
• PySpark for distributed data processing
• SQL for complex query optimization and analytics engineering

Data Architecture & Engineering

• Enterprise Data Warehouse (EDW) architecture
• Dimensional modeling (Star/Snowflake)
• Data Vault 2.0 modeling
• Metadata-driven ingestion frameworks
• Change Data Capture (CDC) implementation
• Medallion architecture (Bronze/Silver/Gold)
• Data lineage, governance, cataloging
• Master Data Management (MDM)

Data Products & Analytics Enablement

• Design and delivery of scalable data products
• Business-aligned semantic layer design
• KPI frameworks & enterprise reporting enablement
• Integration of ERP, SaaS, and operational systems
• Hybrid Lakehouse + EDW architectures

DevOps & Engineering Practices

• CI/CD pipelines (Azure DevOps, GitHub Actions, Bitbucket)
• Infrastructure as Code (Terraform, ARM templates)
• Automated testing (unit, integration, data quality)
• Monitoring, logging, and observability
• Agile/Scrum delivery

Strategic & Leadership Impact

• Principal-level solution architecture design
• Cross-functional stakeholder collaboration
• Technical roadmap planning and execution
• Guidance on data governance frameworks (GDPR, SOX)
• Cloud cost optimization strategies
• Migration from legacy EDWs (Teradata, Oracle, SQL Server) to Lakehouse

Ideal Candidate Profile

• 5-10+ years of hands-on experience in data engineering or data architecture
• Strong expertise in Azure + Databricks ecosystem
• Proven experience building scalable, secure, high-performance data platforms
• Ability to lead design discussions, influence architecture decisions, and mentor teams
• Strong problem-solving mindset and passion for modern data engineering