Senior Data Engineer

Titan Professional Resources is seeking a Senior Data Engineer to join one of our clients. This position will focus on designing, developing, and maintaining scalable data solutions while supporting the modernization of enterprise data platforms. The ideal candidate will have strong SQL and Python skills, hands-on experience with Microsoft Fabric and Azure, and the ability to lead technical initiatives and deliver reliable data solutions.

Senior Data Engineer Responsibilities

• Design, develop, test, deploy, and maintain scalable batch and near-real-time data pipelines across enterprise data sources.

• Build modern data engineering solutions using Microsoft Fabric, including Lakehouse, Warehouse, OneLake, Notebooks, and Dataflows.

• Develop reusable ETL/ELT processes and data transformations using T-SQL, Python, PySpark, and Spark SQL.

• Implement incremental loading, change data capture, schema evolution, data reconciliation, error handling, and recovery processes.

• Design relational, dimensional, and Lakehouse data models to support analytics, reporting, and system integrations.

• Monitor and maintain production data pipelines through logging, alerting, data quality checks, metadata management, and operational support procedures.

• Optimize SQL queries, Spark workloads, pipelines, and storage for performance, scalability, reliability, and cost efficiency.

• Apply security best practices, including role-based access controls, managed identities, secrets management, and secure connectivity.

• Implement source control, automated testing, CI/CD, and deployment processes using Git, Azure DevOps, and Microsoft Fabric.

• Participate in code reviews, release planning, incident resolution, root-cause analysis, and post-implementation reviews.

• Develop reusable engineering standards, templates, frameworks, and technical documentation.

• Collaborate with analysts, Power BI developers, architects, governance teams, and business stakeholders to deliver reliable data solutions.

• Support the modernization and migration of legacy SSIS, SSRS, Azure Data Factory, SQL-based, and other data integration workloads to cloud and Microsoft Fabric platforms.

Senior Data Engineer Qualifications

• Bachelor's degree, college diploma, or equivalent practical experience in Computer Science, Information Systems, Data Engineering, Software Engineering, or a related field.

• Minimum of 5 years of progressive experience in data engineering, data integration, data warehousing, analytics engineering, or a related field.

• Demonstrated experience designing and delivering end-to-end data solutions and supporting production data environments.

• Advanced SQL and T-SQL skills, including complex queries, stored procedures, views, indexing, execution-plan analysis, and performance tuning.

• Strong Python skills for data engineering, automation, testing, API integration, and file processing.

• Hands-on experience with Microsoft Fabric and Azure data services, including Lakehouse, Warehouse, OneLake, Delta tables, pipelines, notebooks, Azure Data Factory, Azure SQL, Azure Data Lake Storage, and Key Vault.

• Experience with PySpark and Spark SQL is preferred.

• Strong understanding of data warehousing, Lakehouse architecture, dimensional modeling, medallion architecture, and ETL/ELT processes.

• Experience with Git, branching, pull requests, automated testing, CI/CD, and deployment management.

• Knowledge of Microsoft Entra ID, managed identities, service principals, role-based access control, and data security best practices.

• Familiarity with data monitoring, logging, alerting, lineage, metadata, data quality, and audit requirements.

• Experience integrating databases, REST APIs, JSON, CSV, Parquet, Delta, SFTP, and other data sources.

• Working knowledge of Power BI and semantic models; DAX experience is a plus.

• Experience with PostgreSQL, SSIS, and SSRS is an asset.

• Strong troubleshooting, analytical, communication, and problem-solving skills.

• Ability to lead technical discussions, mentor engineers, conduct code reviews, and establish engineering best practices.

• Experience in insurance or another regulated financial services environment is strongly preferred.

• Microsoft Fabric, Azure Data Engineer, Power BI, or related Microsoft certifications are a plus.