Data Engineer

Requirements:

• Strong Python programming skills, including experience developing production data pipelines and integrating with APIs.
• Proficiency in SQL for data loading, validation, and troubleshooting.
• Hands-on experience with Snowflake and relevant Azure data and storage services.
• Experience building and operating workflows with Prefect or a comparable orchestration tool, with the ability to work in Prefect.
• Experience ingesting structured and semi-structured data, including formats such as CSV, JSON, and Parquet.
• Understanding of batch and incremental ingestion patterns, pagination, API rate limits, and schema evolution.
• Experience using Git, code reviews, and automated testing.
• Strong troubleshooting, documentation, and communication skills.
• Ability to work independently while coordinating across internal teams and external organizations.

Expected outcomes:

• Reliable, timely ingestion of source data into the raw data layer.
• Maintainable pipelines with clear monitoring, validation, and recovery procedures.
• Documented integrations that can be supported and extended by other engineers.
• Prompt identification and resolution of ingestion issues.

Roles & Responsibilities:

• Develop and maintain data ingestion pipelines using Python, Prefect, Azure, and Snowflake.
• Integrate data from enterprise applications, SaaS tools, and external partners through APIs, database connections, and file transfers.
• Land raw data in designated storage and Snowflake environments, preserving source fidelity and traceability.
• Implement scheduled and incremental ingestion, including retries, error handling, duplicate prevention, and recovery from failed runs.
• Monitor pipeline execution, troubleshoot issues, and resolve data delivery failures.
• Implement ingestion checks for completeness, freshness, schema changes, and reconciliation with source systems.
• Collaborate with business teams, application owners, and external partners to define source interfaces, data formats, delivery schedules, and access requirements.
• Follow established standards for secure access, credential management, logging, and handling sensitive data.
• Maintain pipeline documentation, source mappings, operational runbooks, and support handover materials.

Skills

Python, SQL, Snowflake