Data Engineer - Pharma Commercial Data

We are seeking a skilled Data Engineer with strong expertise in SQL, Python, PySpark, ETL development, and Azure Databricks to design, build, and optimise scalable data pipelines. The ideal candidate should have hands-on experience working with pharmaceutical commercial datasets, including **claims, prescription (Rx), patient, sales, and payer data, and be proficient in processing large-scale healthcare data to support analytics and business intelligence initiatives.

Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using Python, PySpark, and SQL.
• Build and optimise data processing workflows in Azure Databricks.
• Develop and maintain data ingestion frameworks from multiple structured and unstructured data sources.
• Process and transform pharmaceutical commercial datasets, including claims, prescription, payer, sales, formulary, and patient-level data.
• Optimise Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, validation rules, and monitoring processes.
• Collaborate with business analysts, data scientists, and commercial analytics teams to deliver reliable datasets.
• Develop reusable data engineering components and automation frameworks.
• Troubleshoot data pipeline issues and ensure high availability of data platforms.
• Participate in code reviews and follow best practices for version control and CI/CD.
• Ensure compliance with healthcare data governance, privacy, and security standards.

Requirements:

• Strong SQL programming and query optimisation skills.
• Proficiency in Python for data engineering and automation.
• Hands-on experience with PySpark and Apache Spark.
• Experience developing ETL/ELT pipelines.
• Strong experience with Azure Databricks.
• Experience with Azure Data Lake Storage (ADLS).
• Knowledge of Azure Data Factory (ADF) for orchestration.
• Familiarity with Delta Lake, Parquet, and other big data storage formats.
• Experience with Git or Azure DevOps for source control.
• Good understanding of data modelling and data warehousing concepts.
• Strong analytical and problem-solving skills.
• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

Domain Expertise:

• Experience working with pharmaceutical commercial datasets is mandatory.
• Hands-on knowledge of one or more of the following: Medical Claims Data, Pharmacy Claims Data, Prescription (Rx) Data, Patient-Level Data, Sales and Commercial Data, Payer and Formulary Data.
• Understanding of healthcare data standards and commercial analytics workflows is highly preferred.

Preferred Qualifications:

• Experience with Azure Data Factory, Synapse Analytics, or Microsoft Fabric.
• Knowledge of Delta Live Tables and Databricks Workflows.
• Familiarity with data governance and data catalogue tools.
• Experience with performance tuning for Spark workloads.
• Exposure to Agile/Scrum development methodologies.
• Healthcare or Life Sciences industry experience.

Preferred Certifications:

• Microsoft Certified: Azure Data Engineer Associate.
• Databricks Certified Data Engineer Associate/Professional.

Key Competencies:

• Strong analytical and troubleshooting skills.
• Excellent communication and stakeholder management.
• Ability to work independently and collaboratively.
• Attention to detail and commitment to data quality.
• Continuous learning mindset and adaptability.