Senior Data Engineer

Role Summary

We are looking for a highly skilled Senior Data Engineer with strong hands-on experience in Python, SQL, Snowflake, Airflow, ETL/ELT, data warehousing, Fivetran, GitHub, CI/CD, API integration, and data outbound pipelines.

The candidate will be responsible for designing, building, optimizing, and maintaining scalable data pipelines and data warehouse solutions. The role requires strong engineering discipline, production mindset, troubleshooting ability, and experience working with modern cloud data platforms.

The ideal candidate should be able to work independently with architects, data modelers, business analysts, application teams, and downstream consumers to deliver reliable, secure, and high-quality data products.

*Key Responsibilities*

Design, develop, and maintain scalable ETL/ELT data pipelines using Python, SQL, Airflow, Snowflake, and modern data engineering tools.

Build and optimize data ingestion, transformation, validation, reconciliation, and outbound data pipelines.

Develop reusable Python frameworks for data extraction, transformation, quality checks, API calls, file processing, logging, alerting, and error handling.

Create and manage Airflow DAGs for orchestration, scheduling, dependency management, retries, monitoring, and production support.

Work extensively on Snowflake objects including databases, schemas, tables, views, streams, tasks, procedures, stages, file formats, Snowpipe, and role-based access controls.

Develop complex SQL queries, stored procedures, and transformation logic for data warehouse and analytics use cases.

Implement data warehouse design patterns including staging, bronze/silver/gold layers, dimensional models, fact and dimension loading, incremental loads, CDC, SCD handling, and audit frameworks.

Integrate data from multiple sources using Fivetran, APIs, flat files, cloud storage, databases, and enterprise applications.

Design and support data outbound pipelines to send curated data to external systems, downstream applications, APIs, third-party vendors, CRM platforms, operational systems, and reporting platforms.

Set up and maintain GitHub-based development workflows including branching strategy, pull requests, code reviews, version control, and release management.

Implement CI/CD pipelines for automated testing, deployment, environment promotion, code quality checks, and production releases.

Collaborate with data architects and data modelers to implement scalable and business-aligned data warehouse solutions.

Ensure data quality through validation rules, reconciliation checks, duplicate handling, null checks, schema checks, and anomaly detection.

Monitor data pipelines, troubleshoot failures, resolve performance bottlenecks, and support production incidents.

Document technical design, pipeline logic, source-to-target mappings, dependencies, operational procedures, and runbooks.

Follow enterprise standards for security, data governance, access control, data privacy, auditability, and compliance.

Required Skills and Experience

Strong hands-on experience as a Senior Data Engineer

Advanced SQL skills, including joins, window functions, CTEs, query optimization, analytical SQL, stored procedures, and performance tuning.

Strong Python programming skills for data engineering, automation, API integration, file processing, and framework development.

Hands-on experience with Airflow for workflow orchestration, DAG development, monitoring, retries, scheduling, and production operations.

Strong experience with Snowflake including data loading, transformation, performance optimization, warehouse sizing, RBAC, stages, streams, tasks, and Snowpipe.

Strong understanding of ETL/ELT design patterns, data integration, batch processing, incremental loading, CDC, and data reconciliation.

Experience working with Fivetran or similar ingestion tools for source system replication and pipeline management.

Experience with GitHub, version control, branching, pull requests, code reviews, and release management.

Hands-on experience with CI/CD pipelines for data engineering deployments.

Experience building and consuming REST APIs for data ingestion and outbound data exchange.

Good understanding of data warehousing concepts, including dimensional modeling, facts, dimensions, SCD Type 1/2, surrogate keys, and layered data architecture.

Experience in production support, pipeline monitoring, logging, alerting, and failure recovery.

Strong debugging and problem-solving skills across SQL, Python, orchestration, and data platform layers.

Good to Have

Experience with cloud platforms such as Azure

Experience with, Coalesce, or DBT or similar transformation tools.

Experience with API authentication patterns such as OAuth2, JWT, API keys, basic authentication, and token refresh.

Experience with data governance, metadata management, data lineage, and data cataloging tools.

Good to have Experience with containerization, Docker, Kubernetes, or serverless data workloads.

Good to have Exposure to real-time or near-real-time data pipelines using Kafka, Event Hub, streams, or message queues.

Key Technical Areas

Python data engineering

Advanced SQL development

Snowflake development and optimization

Airflow DAG orchestration

ETL/ELT pipeline development

Fivetran ingestion management

Data warehouse implementation

API-based data integration

Outbound data pipelines

GitHub and CI/CD deployment

Data quality and reconciliation

Production monitoring and support

Expected Deliverables

Production-ready ETL/ELT pipelines

Reusable Python data engineering components

Airflow DAGs and orchestration workflows

Snowflake tables, views, procedures, streams, tasks, and loading frameworks

Source-to-target mapping implementation

Data quality and reconciliation checks

Outbound data feeds and API integrations

CI/CD deployment pipelines

Technical design documents and runbooks

Pipeline monitoring, logging, and alerting setup

Qualifications

Bachelor’s degree in Computer Science, Information Technology, Data Engineering, or a related field.

Minimum 6+ years of experience in data engineering, ETL/ELT development, SQL, and data warehousing.

Minimum 3+ years of hands-on experience with Python and cloud data platforms.

Strong communication skills and ability to work with team