Senior Data Engineer

Role Overview :

We are looking for an experienced Senior Data Engineer to design, develop, and optimize scalable, reliable, and secure enterprise data platforms and pipelines.

The role requires strong expertise in dbt, Amazon Redshift, modern data engineering, cloud data platforms, data architecture, data modeling, database technologies, and enterprise data security.

The candidate will be responsible for designing and supporting enterprise dbt environments integrated with Amazon Redshift, including platform administration, user and role configuration, integration with existing development environments, and development of dbt-based transformation solutions.

The role will also support secure access to enterprise data through Natural Language / Conversational Analytics and BI capabilities, ensuring that user authentication, roles, and data-access permissions are consistently enforced across the analytics interface, dbt, and underlying Redshift datasets.

Key Responsibilities :

- Design, develop, and maintain end-to-end data pipelines, data platforms, and integration solutions across batch and streaming workloads.

- Translate business, functional, and technical requirements into maintainable and production-ready data solutions.

- Build scalable data flows across structured, semi-structured, and unstructured data sources.

- Develop data transformation and processing workflows capable of supporting large data volumes and evolving business requirements.

- Design, develop, and maintain transformation models and other components using dbt.

- Configure and administer enterprise-level dbt environments.

- Manage dbt users, roles, authentication, authorization, and role-based access controls.

- Integrate dbt with Amazon Redshift datasets and enterprise data environments.

- Integrate dbt with existing Azure DevOps (ADO) configurations and development workflows.

- Implement data quality, validation, reconciliation, testing, and observability practices across data pipelines.

- Optimize data pipelines and data-processing workloads for performance, scalability, reliability, and cost efficiency.

- Implement appropriate security, privacy, access control, and data governance throughout the data lifecycle.

- Ensure secure and controlled access to sensitive PII and PCI datasets.

- Support Natural Language / Conversational BI solutions connected to dbt and Redshift.

- Ensure conversational analytics platforms honour underlying user credentials, roles, and data-access permissions.

- Monitor pipelines and platform components, troubleshoot failures, perform root-cause analysis, and implement corrective actions.

- Collaborate with architects, business stakeholders, application teams, QA/UAT teams, and other cross-functional stakeholders.

- Contribute to CI/CD practices, automation, reusable frameworks, engineering standards, documentation, mentoring, and continuous improvement initiatives.

Key Requirements :

- 8+ years of relevant experience in Data Engineering or related areas.

- Excellent hands-on knowledge of dbt architecture, configuration, administration, and user setup.

- Experience with enterprise-level dbt installation and configuration.

- Strong hands-on experience with dbt development, including transformation models and associated dbt components.

- Strong experience integrating dbt with Amazon Redshift.

- Hands-on experience configuring role-based access controls within dbt.

- Strong understanding of authentication and authorization across dbt and underlying Redshift datasets.

- Experience integrating dbt with Azure DevOps (ADO) and enterprise development workflows.

- Strong proficiency in SQL, including complex queries, analytical functions, stored procedures, and query/performance optimization.

- Strong understanding of modern data architectures including :

1. Data Lake

2. Lakehouse

3. Cloud Data Warehouse

4. Medallion Architecture

5. ELT patterns

- Strong knowledge of data warehousing and data modeling concepts including:

1. Dimensional modeling

2. Fact and dimension design

3. Slowly Changing Dimensions (SCD)

- Hands-on experience with cloud data platforms and modern data engineering technologies.

- Experience working with relational and non-relational databases.

- Experience with data ingestion and streaming technologies.

- Strong understanding of data quality, governance, security, and observability practices.

- Proficiency in Python, Shell, PowerShell, or equivalent scripting and automation technologies.

- Experience with Git, CI/CD, and Agile delivery practices.

- Experience implementing or supporting Natural Language / Conversational BI capabilities will be highly preferred.

- Strong understanding of security requirements for PII and PCI data.

- Strong analytical, troubleshooting, and problem-solving capabilities.

- Excellent communication, documentation, collaboration, and stakeholder management skills.

- Strong ownership and accountability for production-grade data solutions.

Skills

Data Engineering, Data Build Tool, Redshift, Python, SQL, ETL, NLP, PowerShell, Data Ingestion, Scripting