Sr. Data and Analytics Operations Engineer

A Data and Analytics Operations Engineer (often called a DataOps Engineer) is a specialised role focused on the "production line" of data. They bridge the gap between data engineering (which builds the systems) and data analysis (which uses the data), ensuring that pipelines are automated, reliable, and secure.

Responsibilities:

• Pipeline Monitoring and Maintenance: Assist in the daily operation of data pipelines using Azure Data Factory (ADF) and Databricks. This includes troubleshooting job failures, monitoring performance bottlenecks, and ensuring data flows reliably into the Delta Lake.
• Data Quality and Integrity: Perform data mapping and quality assurance checks to identify inconsistencies. You will often assist in developing "gold-layer" data products, structured, analysis-ready datasets that meet enterprise standards.
• Code and Automation: Write and optimise Spark SQL and PySpark code within Databricks notebooks to automate manual data transformations.
• Documentation: Create and maintain technical documentation for data models, pipeline workflows, and data dictionaries to support team-wide transparency.
• Collaborative Support: Work closely with senior engineers, vendor partners and stakeholders to translate business requirements into technical solutions and assist in change management documentation.
• Contribute to technical documentation and solution design assets.

Requirements:

• Platforms: Hands-on familiarity with Azure Databricks, Azure Data Lake Storage (ADLS), and Azure Data Factory.
• Languages: Proficiency in SQL (complex querying) and Python/PySpark (data transformation), job clusters, Unity Catalogue, and Databricks Notebooks.
• Tools: Basic experience with Git for version control and familiarity with CI/CD workflows via Azure DevOps. Excellent communication skills and root-cause analysis.
• Experience: 1-3 years of professional experience, including internships or academic projects involving cloud data services.
• Education: A bachelor's degree in computer science, data science, statistics, or a related quantitative field.

Preferred Certifications:

• Microsoft Certified: Azure Data Engineer Associate or Azure Fundamentals.
• Databricks Certified Professional Data Engineer.

Skills

azure databricks, databricks, pyspark, python, sql