Senior Microsoft Fabric Data Engineer

Job Description :

We are looking for a Senior Microsoft Fabric Data Engineer with strong hands-on experience in Microsoft Fabric Data Engineering, SQL, PySpark, Data Warehousing, and Data Modelling.

The ideal candidate will be responsible for designing, developing, and optimizing scalable data solutions using the Microsoft Fabric ecosystem.

The candidate should have strong expertise in building modern data pipelines, transforming and processing large datasets, developing robust data models, and implementing efficient data warehouse solutions.

A good understanding of Power BI/BI concepts, Azure DevOps, Git-based development practices, and CI/CD workflows is also required.

The role requires an individual with strong logical and analytical thinking, excellent problem-solving capabilities, attention to data quality, and the ability to communicate effectively with technical and business stakeholders.

Key Responsibilities :

- Design, develop, and maintain scalable data engineering solutions using Microsoft Fabric.

- Build and optimize data pipelines using Microsoft Fabric Data Factory, Data Pipelines, Notebooks, Lakehouse, and related Fabric components.

- Develop efficient data transformation and processing solutions using PySpark and SQL.

- Work extensively with large and complex datasets, ensuring data is processed efficiently and reliably.

- Design and implement Data Warehouse and Lakehouse architectures aligned with business and technical requirements.

- Develop robust data models, including conceptual, logical, and physical data models.

- Implement dimensional modelling concepts such as fact tables, dimension tables, star schemas, and slowly changing dimensions (SCD) where applicable.

- Write complex and optimized SQL queries, stored procedures, views, and data transformation logic.

- Develop PySpark notebooks for data ingestion, cleansing, transformation, aggregation, and enrichment.

- Implement data validation, reconciliation, quality checks, and error-handling mechanisms across data pipelines.

- Monitor pipeline performance and troubleshoot data processing and integration issues.

- Optimize SQL queries, PySpark jobs, data models, and pipeline execution for improved performance and scalability.

- Work with structured and semi-structured data from multiple sources and integrate them into centralized data platforms.

- Collaborate with BI developers and analysts to provide reliable, well-modelled datasets for reporting and analytics.

- Have a good understanding of Power BI and BI reporting concepts, including datasets, semantic models, measures, and reporting requirements.

- Translate business requirements into scalable data engineering and modelling solutions.

- Participate in technical discussions, architecture/design reviews, and solution planning.

- Follow software engineering best practices for version control, code quality, documentation, and deployment.

- Use Git and Azure DevOps for source-code management, branching, commits, pull requests, and code reviews.

- Work with VS Code and other development tools for data engineering development.

- Participate in peer code reviews and ensure that development follows established coding and engineering standards.

- Collaborate with cross-functional teams including data architects, BI developers, analysts, application teams, and business stakeholders.

- Prepare technical documentation related to data pipelines, data models, transformation logic, and solution architecture.

Required Technical Skills :

- Strong hands-on experience with Microsoft Fabric Data Engineering.

- Experience working with Fabric Lakehouse, Data Factory/Data Pipelines, Notebooks, and data processing capabilities.

- Understanding of modern data platform and Lakehouse architecture.

- Experience designing and implementing production-grade data pipelines in Fabric.

- Strong proficiency in SQL.

- Experience writing complex queries, joins, CTEs, subqueries, window functions, aggregations, and transformations.

- Strong understanding of SQL optimization and query performance.

- Experience working with relational databases and analytical data stores.

- Strong hands-on experience with PySpark for large-scale data processing.

- Experience developing notebooks for data ingestion, transformation, cleansing, and aggregation.

- Understanding of Spark execution, partitions, joins, caching, and performance optimization.

- Ability to troubleshoot and optimize PySpark workloads.

- Strong understanding of Data Warehouse concepts and architecture.

- Hands-on experience with dimensional modelling.

- Good understanding of fact and dimension tables, star/snowflake schemas, surrogate keys, and SCD concepts.

- Ability to design scalable and performance-oriented data models.

- Experience working with enterprise data and analytical workloads.

- Good understanding of Power BI and BI concepts.

- Ability to understand reporting and analytics requirements and provide appropriate curated datasets.

- Understanding of semantic models, measures, data relationships, and BI consumption patterns is desirable.

- Experience with Azure DevOps and Git.

- Strong understanding of Git workflows, branching strategies, commits, pull requests, merges, and conflict resolution.

- Experience participating in code reviews and peer reviews.

- Familiarity with CI/CD concepts and deployment practices.

- Hands-on experience using VS Code or similar development environments.

Required Soft Skills :

- Strong logical and analytical thinking.

- Excellent problem-solving and troubleshooting abilities.

- Strong attention to data accuracy, quality, and consistency.

- Ability to understand complex business requirements and translate them into technical solutions.

- Excellent written and verbal communication skills.

- Ability to collaborate effectively with technical and non-technical stakeholders.

- Strong ownership and accountability for deliverables.

- Ability to work independently as well as within a collaborative team environment.

- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related discipline.

- Microsoft certifications related to Azure Data Engineering, Microsoft Fabric, or Power BI would be an advantage.

- Experience working on enterprise-scale data engineering or analytics projects.

- Exposure to cloud-based data platforms and modern data architecture patterns.

Skills

Microsoft Fabric, Data Engineering, PySpark, Data Warehousing, Data Pipeline, Azure Data Factory, SQL, Data Ingestion, Dimensional Modeling, Power BI