Senior Data Engineer

Job Description: Snowflake Data Engineer - Python, PySpark & AWS

Job Overview

We are seeking a skilled Snowflake Data Engineer with strong expertise in Snowflake, Python, PySpark, and SQL to design, develop, optimize, and maintain scalable data pipelines and data warehousing solutions. The ideal candidate will have hands-on experience with AWS, GitHub, and modern data engineering practices, along with a good understanding of machine learning, AI development tools, and emerging software development methodologies.

The candidate should possess strong analytical and problem-solving skills, a solid understanding of ETL/ELT processes, and the ability to collaborate with cross-functional teams to deliver high-quality, reliable, and efficient data solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT processes and data pipelines using Python, PySpark/Snowpark, SQL, and Snowflake.
• Build, optimize, and manage data processing workflows and data transformation solutions on AWS, including Amazon S3.
• Develop efficient SQL queries, stored procedures, and data transformation logic, applying performance optimization techniques.
• Implement and maintain data warehousing solutions using dimensional modeling, normalized data structures, and industry-standard data modeling practices.
• Work with GitHub and related development tools to manage source code, collaborate with team members, and support version control and CI/CD processes.
• Develop and maintain data quality checks, monitoring mechanisms, and error-handling processes to ensure data accuracy, consistency, and reliability.
• Collaborate with technical teams, business stakeholders, and internal and external technology resources to gather requirements and finalize technical specifications.
• Write, review, and maintain detailed technical specifications for complex system components.
• Troubleshoot and resolve complex technical issues, perform root-cause analysis, and implement effective bug fixes.
• Apply critical thinking and research skills to identify opportunities for improving data processing efficiency, scalability, and system performance.
• Leverage AI development tools and explore modern development approaches, including Spec-Driven Development (SDD) and AWS AI-DLC.
• Contribute to Agile or Waterfall development processes, ensuring adherence to coding standards, development best practices, and quality expectations.
• Support the integration of machine learning models and statistical analysis techniques into data engineering workflows, where applicable.

Required Skills and Qualifications

• Strong hands-on expertise in Snowflake, Python, PySpark, and SQL.
• Experience with PySpark and/or Snowpark for large-scale data processing and transformation.
• Proven experience developing and maintaining ETL/ELT workflows and data pipelines.
• Hands-on experience with AWS services, particularly Amazon S3.
• Practical experience using GitHub for version control and collaborative development.
• Strong understanding of relational databases, data warehousing concepts, and ETL processes.
• Advanced SQL skills, including query optimization, performance tuning, and complex data transformations.
• Knowledge of normalized and dimensional data modeling principles.
• Experience with data quality, validation, monitoring, and error-handling practices.
• Familiarity with software development methodologies, including Agile and Waterfall.
• Experience with machine learning models or statistical analysis.
• Excellent analytical, critical-thinking, debugging, and problem-solving skills.
• Strong communication and collaboration skills, with the ability to work effectively with technical teams and stakeholders.
• Ability to understand complex requirements, contribute to technical design, review specifications, and deliver reliable software components.

Preferred / Good-to-Have Skills

• Experience with Apache Iceberg and modern open table formats.
• Knowledge of Apache Airflow or other workflow orchestration tools.
• Experience with Terraform or other Infrastructure as Code (IaC) tools.
• Familiarity with GitHub Actions and CI/CD pipeline implementation.
• Knowledge of advanced data modeling and data warehousing techniques.
• Familiarity with AI-assisted software development tools.
• Understanding of Spec-Driven Development (SDD) and AWS AI-DLC.
• Exposure to machine learning workflows and statistical analysis in data-intensive environments.
• Hands-on experience with Jetstream.

Primary Skills

• Python
• PySpark / Snowpark
• SQL
• Snowflake
• ETL/ELT and Data Pipeline Development
• AWS / Amazon S3

Secondary Skills

• Apache Iceberg
• Apache Airflow
• Terraform / Infrastructure as Code
• GitHub Actions / CI/CD
• Data Modeling and Data Warehousing
• Data Quality and Monitoring
• AI Development Tools
• Spec-Driven Development (SDD) / AWS AI-DLC

Our Commitment to Inclusion

We welcome applications from individuals of all backgrounds, including underrepresented groups, people with disabilities, career changers, and candidates with non-traditional education or experience.

If you require accommodations at any stage of the recruitment process, please let us know. We are committed to fostering an inclusive and supportive environment where everyone has the opportunity to succeed.