Data Engineer

• ·         Develop and maintain data ingestion pipelines from enterprise systems into the Databricks bronze layer.

·         Use Azure Data Factory and the existing integration framework to onboard and manage data sources efficiently.
·         Ensure bronze-layer data is clean, well-structured, and documented, so it can be reliably reused by BI and AI teams downstream.
·         Implement data-quality checks, reconciliation, monitoring, and issue resolution for the bronze layer.
·         Maintain and support the setup and infrastructure of the dbt framework used by downstream teams for silver and gold layer transformations.
·         Implement and maintain data governance practices, including access controls, data classification, and lineage, to protect PII and sensitive data.
·         Support simplified and governed data-access processes for authorized users.
·         Develop automation that improves secure and efficient user interaction with data, such as self-service access-request workflows with automated approval and provisioning, or automated PII detection and classification during ingestion.
·         Optimize Databricks compute, jobs, storage, and queries for performance, reliability, and cost efficiency.
·         Maintain technical documentation, operational runbooks, and data-pipeline standards.
·         Work closely with BI, AI, IT, source-system owners, and business stakeholders.

Qualifications

• ·         Bachelor’s degree in Computer Science, IT, Data Engineering, or a related field; equivalent experience is welcome.

·         3–6 years of experience in data engineering, ETL/ELT, analytics engineering, or database development.
·         Strong SQL skills, including transformation logic, data validation, and performance tuning.
·         Hands-on experience applying data and query optimization techniques, such as partitioning, clustering, file compaction, caching, and efficient join strategies, to improve pipeline performance and reduce compute cost.
·         Experience with data pipeline development, orchestration, scheduling, and monitoring.
·         Experience integrating data from multiple systems, such as databases, APIs, files, and enterprise applications.
·         Understanding of lakehouse/medallion architecture, data modeling, and batch data processing.
·         Experience with DevOps/DataOps practices, including CI/CD, version control, and automated testing for data pipelines.
·         Practical experience with data governance concepts, including access controls, data classification, and lineage.
·         Experience with Git and collaborative development practices.
·         Strong analytical, troubleshooting, and communication skills.

Preferred Experience:

• ·         Experience with Databricks Unity Catalog and centralized data-access governance.

·         Python or PySpark experience.
·         Hands-on experience with Databricks, Azure Data Factory, Apache Spark, Delta Lake, Azure Data Lake Storage, Azure Key Vault, Azure DevOps, and Microsoft Entra ID.
·         Experience with github or Azure DevOps to develop CI/CD, automated testing, monitoring, and alerting for data pipelines.
·         Working knowledge of dbt, including model structure, testing, and documentation, sufficient to maintain and support the framework’s setup.
·         Exposure to healthcare data domains, such as electronic medical records (EMR/EHR), claims, or clinical operational data.
·         Understanding of healthcare data privacy regulations, such as Thailand's PDPA or HIPAA, and their application to data engineering practices.