Lead Data Engineer

The Phoenix Group is hiring a Lead Data Engineer to own the design, development, and maintenance of enterprise data architecture and pipelines. The role supports analytics, AI/ML, and data governance initiatives while ensuring secure, high-quality, and performant data solutions.

Responsibilities

• Design, build, test, deploy, and support scalable data pipelines utilizing orchestration tools (Airflow, Astronomer), data transformation frameworks (dbt), cloud data platforms (Snowflake), SQL, and Python
• Develop and optimize cloud-based data architecture, including database schemas, data warehouses, security roles, workload isolation, cost management, and performance tuning within Snowflake
• Implement medallion data architecture principles—raw/bronze, silver, and gold layers—for structured and semi-structured data management
• Lead migration and modernization of legacy workflows to cloud orchestration platforms, establishing reusable patterns for dependency management, retries, backfills, alerting, recovery, and ensuring idempotency
• Troubleshoot and maintain the full runtime environment of orchestration platforms, including schedulers, workers, executors, metadata stores, queues, and autoscaling components
• Collaborate with cloud infrastructure teams to manage AWS services such as S3, IAM, EKS, ECS, Fargate, EC2, Lambda, Secrets Manager, KMS, CloudWatch, and networking components
• Build secure, resilient integrations across APIs, SFTP, databases, cloud storage, vendor APIs, flat files, Excel files, and event-driven systems
• Develop ingestion and processing pipelines for structured, semi-structured, and unstructured data, including OCR/image-based document extraction
• Embed data quality testing, monitoring, lineage, reconciliation, and observability into data workflows to ensure reliability and compliance
• Partner with Data Governance, Security, and Business teams to enforce data ownership, classification, access policies, retention, and auditing standards
• Build and maintain CI/CD pipelines for orchestration workflows, transformations, and infrastructure code, ensuring automated testing, environment promotion, deployment, and rollback capabilities
• Support production environments through on-call and incident management processes, conducting root cause analysis, stakeholder communication, and corrective actions
• Establish reusable frameworks, engineering standards, and best practices to improve platform reliability, consistency, and delivery velocity
• Lead technical workstreams, mentoring engineering team members through design reviews, code reviews, documentation, and troubleshooting

Skills

• Minimum of 5+ years in data engineering or software engineering roles with progressive responsibilities
• Proven experience with cloud-based data platforms, including Snowflake, AWS cloud infrastructure, and data orchestration tools such as Airflow/Astronomer
• Strong expertise in designing and implementing data architectures, including dimensional and medallion models
• Hands-on experience migrating and maintaining production workflows in orchestration platforms
• Deep understanding of cloud compute environments, including Kubernetes, containers, schedulers, workers, and autoscaling architectures
• Proficiency in AWS services such as S3, IAM, EKS/ECS, Lambda, Secrets Manager, KMS, and CloudWatch for security, storage, and monitoring
• Skilled in integrating complex data sources and systems, including APIs, SFTP, relational and NoSQL databases, cloud storage, and file-based workflows
• Experience processing semi-structured and unstructured data, including OCR, document extraction, and image processing
• Strong background in implementing data quality controls, automated testing, data lineage, and observability tools
• Knowledge of data governance principles, access controls, data classification, retention policies, and audit requirements
• Experience with CI/CD pipelines, Git workflows, and automated deployment practices for data solutions and infrastructure
• Excellent troubleshooting skills across cloud infrastructure, data pipelines, application code, and security permissions
• Ability to lead technical initiatives and communicate effectively with both technical teams and business stakeholders

Company Overview