Senior Data Engineer

Job Description:

The Senior Data Engineer will be responsible for designing, developing, and maintaining scalable cloud-based data platforms and data pipelines. The role requires strong expertise in AWS, Snowflake, Databricks, Python, SQL, and PySpark, along with experience in batch and real-time data processing. The candidate will collaborate with cross-functional teams to build reliable, secure, and highperformance data solutions that support analytics, reporting, and AI/ML initiatives.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines and ETL/ELT frameworks.
• Build and optimize batch and real-time streaming solutions using PySpark and cloud-native services.
• Develop and manage enterprise data platforms using AWS, Snowflake, and Databricks.
• Design and implement robust data models, data lakes, data warehouses, and lakehouse architectures.
• Integrate data from multiple sources while ensuring data quality, governance, and security.
• Optimize performance and scalability of data processing workloads.
• Collaborate with Data Scientists, Analysts, Architects, and business stakeholders to deliver data solutions.
• Implement monitoring, alerting, and automated data validation processes
• Contribute to architecture reviews, technical documentation, and best practices.
• Mentor junior engineers and participate in code reviews.
• Stay updated on emerging cloud and data engineering technologies.

Required Skills and Experience:

• 5-8 years of relevant experience in Data Engineering, Big Data, and Cloud Data Platform development.
• Strong proficiency in Python, SQL, and PySpark.
• Hands-on experience designing, developing, and maintaining scalable data platforms on AWS Cloud.
• Strong expertise in Snowflake and Databricks for enterprise-scale data engineering solutions.
• Experience building and optimizing batch and real-time streaming data pipelines.
• Strong understanding of Data Modeling concepts, including Dimensional
• Modeling, Star Schema, Snowflake Schema, and Data Vault.
• Experience designing and implementing ETL/ELT pipelines for large-scale enterprise data platforms.
• Strong expertise in Apache Spark and distributed data processing frameworks.
• Experience building and managing Data Lakes, Data Warehouses, and Lakehouse architectures.
• Hands-on experience with AWS services such as S3, Glue, EMR, Lambda, Redshift, Athena, EC2, ECS/EKS, Kinesis, MSK, Step Functions, CloudWatch, IAM, RDS, DynamoDB, EventBridge, SNS, and SQS.
• Experience integrating data from multiple structured and unstructured data sources.
• Strong understanding of data governance, security, data quality, and performance optimization best practices.
• Experience with version control systems such as Git and collaborative development workflows.
• Proficiency with CI/CD pipelines and software engineering best practices.
• Strong problem-solving skills with the ability to debug, troubleshoot, and resolve complex data engineering challenges.
• Ability to design scalable and maintainable data solutions using modern architectural patterns and best practices.
• Proven ability to work collaboratively in cross-functional teams, including participation in code reviews, technical discussions, and knowledge-sharing initiatives.
• Experience working in Agile/Scrum delivery environments.
• Strong communication skills and ability to interact with business and technical stakeholders.
• Experience in Healthcare, Pharma, or Real-World Evidence (RWE) data environments (good to have).
• Experience with Infrastructure as Code (Terraform/CloudFormation) (good to have).
• Experience with Docker and Kubernetes (good to have).
• Knowledge of Data Mesh, CDC, and Event-Driven Architectures (good to have).
• Exposure to MLOps and AI/ML platform integration (good to have).
• AWS, Snowflake, or Databricks certifications (good to have).