Senior Data Engineer

As the Lead, Data Engineering, you will play a pivotal role in designing and prototyping cutting-edge solutions that empower our operations with timely and relevant data. You'll explore new tools and build cost-effective solutions using the powerful Snowflake and AWS Data Platform. In this position, you will also lead and inspire our cross-vendor Data Engineering team, driving key initiatives that shape the future of our organisation.

Responsibilities:

• Develop, customise, and manage integration tools, databases, warehouses, and analytical systems.
• Develop data pipelines from internal and external data sources to support new product launches and drive data quality across data products.
• Build automation and monitoring frameworks that capture metrics and operational KPIs for data pipeline quality and performance.
• Assist in implementation of best practices around systems integration, security, performance and data management.
• Partner with the business, IT, and technical subject matter experts to develop robust solutions.
• Implement DevOps, DataOps and Agile methodologies to improve KPIs like cycle times, consistency, and quality.
• Monitor the overall performance and stability of the system; adjust and adapt automated pipeline as data, models, and/or requirements change.
• Test the reliability and performance of data engineering pipelines and support the testing team with data validation activities.
• Partner with various Snowflake Engineering subject matter experts, including vendor partners, project managers and business teams, to scope and build customer-facing content, modules, tools and proof of concepts.
• Research and cultivate state-of-the-art data engineering methodologies to develop into a Snowflake technology stack developer.

Requirements:

• Bachelor's degree in engineering, computer science, or related field.
• 3 - 5 years of total work experience, with at least 4 years of experience in Python /Snowflake Data Engineering tool stack.
• At least one key data engineering professional certification (e. g., SnowPro Associate and Core).
• Experience with data management tools such as Airflow, Airbyte and DBT.
• Good scripting, data modelling, and programming skills.
• Understanding of cloud architecture principles and best practices.
• Experience in the design and build of end-to-end solutions that meet business requirements and adhere to scalability, reliability, and security standards.
• Familiarity with version control systems such as GitHub, GitHub Actions and DevOps practices for CI/CD pipelines.

Desired Qualifications:

• Subject matter expert for Snowflake Data, Analytics & AI with experience Snowflake architecture design is preferred.
• Good understanding of migration and modernisation strategies and approaches is preferred.
• Experience with Python, EKS Implementations, Spark, Kafka, and/or Flask is preferred.

Skills

airbyte, airflow, dbt, python, snowflake