Lead Data Engineer

Responsibilities:

• Lead and contribute to the development of a self-service data platform for reporting and analytics.
• Design and build data solutions using Databricks, SQL, Python, Spark, and Delta Lake within the Azure ecosystem (Blob Storage, Data Factory, Event Hubs).
• Implement best practices for ETL/ELT processes, focusing on data quality management, partitioning, maintainability, and reusability.
• Collaborate with engineers, product managers, and business leaders to ensure seamless integration of the data platform with systems such as Tableau, Power BI, APIs, and other applications.
• Establish CI/CD pipelines and use infrastructure-as-code tools for deployment and monitoring (Git, Terraform, Azure DevOps/GitHub Actions/Jenkins, Azure Monitor/Datadog).
• Ensure adherence to organizational compliance standards and the Code of Conduct as defined in LogixGarden.

Requirements:

• 8+ years of data engineering experience.
• 3+ years in a senior or principal engineer role, with proven experience leading projects.
• Strong hands-on experience with Databricks and related integrations (Delta Live Tables, Auto Loader, Databricks Workflows / Apache Airflow, Unity Catalogue).
• Expertise in PySpark and/or Scala.
• Strong DataFrame programming experience (Spark, Pandas, NumPy).
• Proven ability to design scalable, secure, and maintainable data pipelines.
• Excellent communication and leadership skills, with a passion for mentoring.
• Solid understanding of data security and governance frameworks, with exposure to HIPAA or SOC 2 compliance preferred.

Preferred Skills:

• AWS cloud experience.
• AWS Databricks implementation expertise.
• Experience designing and implementing data security and governance frameworks.

Skills

databricks, elt, etl, pipelines, pyspark, scala, spark