Data Engineer – Data Warehouse

1. Introduction

This proposal outlines the engagement for Offshore Data Engineering Support to manage and maintain the on-premises Data Warehouse and contribute to the development of business requirements in the Cloud Data Warehouse.

The assigned resources will focus on monitoring, troubleshooting, and resolving production issues in the on-premises environment while also building data models and pipelines in the cloud using Azure Data Factory (ADF).

2. Scope of Work

On-Premise Data Warehouse Support

• Monitor and re-run failed data pipelines and workflows to ensure seamless operations on SSIS & T-SQL.
• Troubleshoot and resolve production issues, focusing exclusively on bug fixes.
• Maintain data accuracy and integrity across all pipelines.
• Conduct root cause analysis for recurring failures and document resolutions.
• Provide daily status reports on pipeline execution, highlighting failures, root causes, and applied fixes.
• Collaborate with onshore teams to address critical production incidents.
• Implement temporary workarounds to minimize downtime while working toward permanent solutions.

Cloud Data Warehouse Development

• Gather and analyze business requirements for business datasets in the cloud.
• Design, develop, and test data pipelines to support Cloud Data Warehouse initiatives.
• Build and optimize batch processing and streaming jobs to ensure timely data refresh.
• Implement Medallion Architecture and design models for structured data processing across different layers.
• Collaborate with onshore teams to establish best practices and reduce technical debt.
• Define data models and ETL strategies in alignment with stakeholder requirements.
• Participate in regular stand-ups and discussions with onshore and offshore teams to ensure seamless collaboration.
• Ensure compliance with data governance policies, security standards, and best practices.
• Develop and maintain comprehensive runbooks to document cloud operations, ensuring clear, step-by-step guidance for troubleshooting and maintenance.
• Standardize and optimize cloud workflows by creating detailed documentation that enhances efficiency, reduces downtime, and supports seamless knowledge transfer across teams.