Lead II - Data Engineering

Role: Sr Backend Developer - Glue, Python, SQL Key Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using AWS Glue (PySpark and Python shell jobs), Glue Data Catalog, crawlers, and workflows
• Write complex, optimized SQL for data transformation, validation, and analysis across platforms such as Amazon Redshift, Athena, and relational databases
• Build reusable Python libraries, frameworks, and utilities for data ingestion, transformation, and quality checks
• Model data for analytical and operational use cases (star/snowflake schemas, data lakes/lakehouse patterns)
• Implement data quality, lineage, monitoring, and ing for production pipelines
• Optimize job performance and cost (partitioning, file formats such as Parquet, job bookmarks, DPU tuning)
• Collaborate with cross-functional teams to gather requirements and translate them into technical designs
• Participate in code reviews, enforce engineering best practices, and mentor junior engineers
• Support CI/CD, infrastructure-as-code, and release processes for data workloads
• Troubleshoot production issues and drive root-cause analysis
• 6+ years of software or data engineering experience, with 3+ years building pipelines on AWS
• Strong hands-on expertise with AWS Glue (ETL jobs, Data Catalog, crawlers, triggers, workflows) and PySpark
• Advanced SQL skills, including query optimization, window functions, CTEs, and performance tuning on large datasets
• Proficiency in Python, including packaging, testing (pytest), and writing maintainable, well-documented code
• Experience with related AWS services: S3, Lambda, Athena, Redshift, Step Functions, IAM, CloudWatch
• Solid understanding of data warehousing concepts, data modeling, and lakehouse architectures
• Experience with version control (Git) and CI/CD pipelines
• Strong problem-solving skills and ability to work independently in an agile environment