Data Engineer

Microsoft Azure Data Engineer to join a growing data and analytics team. The ideal candidate will have strong hands-on experience with Microsoft Azure data services, Azure Synapse Analytics and/or Microsoft Fabric, data engineering, data integration, data transformation, and enterprise data platforms. The role is responsible for designing, developing, implementing, and optimizing scalable data pipelines, data warehouses, lakehouses, and enterprise data solutions using the Microsoft Azure data ecosystem. This role will work closely with data architects, Power BI developers, analysts, software engineers, DevOps engineers, and business stakeholders to build reliable, scalable, secure, and high-performing enterprise data platforms.

Responsibilities:

• Design, develop, and maintain enterprise data solutions using Microsoft Azure.
• Build scalable data ingestion, transformation, and processing solutions.
• Develop and optimize data pipelines using Azure Data Factory, Azure Synapse Analytics, and/or Microsoft Fabric.
• Design solutions that integrate data from databases, APIs, files, SaaS platforms, cloud applications, and enterprise systems.
• Develop reusable data engineering frameworks and components.
• Implement batch and near-real-time data processing solutions.
• Monitor, troubleshoot, and optimize Azure data workloads.
• Support modernization and migration of legacy data platforms to Azure.
• Design and develop data solutions using Azure Synapse Analytics and/or Microsoft Fabric.
• Build enterprise data warehouses and Lakehouse solutions.
• Develop data pipelines and orchestration workflows.
• Work with Synapse Pipelines, Fabric Data Factory, and Azure Data Factory.
• Develop notebooks using Python, PySpark, and Spark SQL.
• Build and optimize data transformation processes.
• Work with Synapse SQL, Fabric Data Warehouse, Fabric Lakehouse, and related technologies.
• Support migration and modernization initiatives between traditional Azure architectures, Azure Synapse, and Microsoft Fabric.
• Evaluate appropriate technologies based on performance, scalability, cost, security, and business requirements.
• Design and develop robust ETL and ELT pipelines.
• Ingest structured, semi-structured, and unstructured data from multiple enterprise sources.
• Integrate data from databases, APIs, files, SaaS applications, and cloud platforms.
• Develop reusable ingestion and transformation frameworks.
• Implement incremental data loads and Change Data Capture (CDC) strategies.
• Develop data validation, reconciliation, and error-handling processes.
• Troubleshoot pipeline failures and data integration issues.
• Optimize pipelines for performance, scalability, reliability, and cost.
• Design and implement modern Data Lake, Lakehouse, and Data Warehouse architectures.
• Work with Azure Data Lake Storage Gen2 and Microsoft OneLake.
• Implement Bronze, Silver, and Gold data layers.
• Design dimensional data models using Star and Snowflake schemas.
• Develop fact and dimension tables.
• Implement Slowly Changing Dimensions (SCD).
• Work with Delta Lake and related technologies.
• Implement partitioning and performance optimization strategies.
• Support enterprise data warehouse modernization and migration initiatives.
• Develop integrations between Azure services and enterprise applications.
• Implement secure and scalable cloud data architectures.
• Support cloud migration and modernization initiatives.
• Monitor Azure resources and optimize platform performance and cost.
• Apply cloud architecture and engineering best practices across data solutions.
• Develop solutions using Fabric Lakehouse and Data Warehouse.
• Build pipelines using Fabric Data Factory.
• Develop Dataflows Gen2.
• Work with OneLake.
• Develop Fabric notebooks using PySpark, Python, and Spark SQL.
• Implement Delta Lake-based data architectures.
• Support Direct Lake and Power BI integration.
• Manage Fabric workspaces and deployment processes.
• Monitor and optimize fabric capacity utilization.
• Develop scalable data transformations using Python, PySpark, and Spark SQL.
• Process large datasets using distributed computing.

Requirements:

• 6 to 10 years of relevant experience in Azure Data Engineering.
• Minimum 6 years of hands-on experience with Azure Data Engineering components, including PySpark, SQL, Azure Data Factory, and Azure Data Services.
• Minimum 2 years of experience with Azure Synapse Analytics or Microsoft Fabric.
• Strong experience in data engineering, data integration, data transformation, and enterprise data platforms.
• Ability to work independently with minimal supervision and contribute through hands-on development.
• Experience with pipeline migration using PySpark and SQL.
• Experience working with Azure technologies, including Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage Gen2, Azure SQL Database, Azure SQL Managed Instance, Azure Functions, Azure Key Vault, Azure Event Hubs, Azure Storage, Microsoft Entra ID, and Microsoft Fabric.

Nice to Have:

• Experience across both Azure Synapse Analytics and Microsoft Fabric.
• Experience with Power BI integration and Direct Lake.

Skills

Azure, Azure Synapse, Fabric, Microsoft Azure, Microsoft Fabric, PySpark, Spark, Synapse