Data/ETL Engineer

About the Role"

"Data Engineer"

Hybrid -Bangalore

Years of EXP - 6+years

The below mentioned is the illustrated job description, that we should refer to for a Data Engineer Profile, (highlighted are the key skills we need)-

The ideal candidate will have expertise in Glue, Data Catalog, Athena, DynamoDB, Lambda, S3, RDS (PostgreSQL), ECS, SQS, SNS, CloudFront, and IAM roles. Strong proficiency in Python, PySpark, Terraform, and Terragrunt, coupled with experience in Agile environments and GitHub, is essential. You will play a pivotal role in building robust data pipelines and infrastructure to support analytics and business intelligence initiatives.

Required Skills and Qualifications

• AWS Expertise:
• Proficiency in AWS Glue, Data Catalog, Athena, DynamoDB, Lambda, S3, RDS (PostgreSQL), ECS, SQS, SNS, and CloudFront.
• Strong understanding of IAM roles and security best practices.
• Programming and Frameworks:
• Advanced proficiency in Python and PySpark for data processing.
• Solid understanding of SQL for querying relational and non-relational databases.
• Infrastructure as Code(good to have)
• Hands-on experience with Terraform and Terragrunt for infrastructure automation.
• Development Practices:
• Experience in Agile methodologies (Scrum, Kanban) and version control systems like GitHub.
• Familiarity with CI/CD pipelines and DevOps practices.
• Other Skills:
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.
• Ability to work independently and in a team-oriented environment.

Key Responsibilities

• Data Engineering and Processing:

• Develop, deploy, and manage scalable ETL/ELT pipelines using AWS Glue and PySpark.
• Work with AWS Data Catalog for metadata management and data discovery.
• Optimize query performance and data analysis workflows using AWS Athena.
• Implement serverless data workflows using AWS Lambda for real-time and batch data processing.

• Database and Storage Management:

• Design and manage relational databases on AWS RDS (PostgreSQL) and non-relational databases on DynamoDB.
• Store, manage, and optimize data in AWS S3 for performance, security, and cost efficiency.

• Infrastructure as Code (IaC) (good to have):

• Build and deploy AWS infrastructure using Terraform and Terragrunt.
• Automate the provisioning of scalable, secure, and reliable resources.

• Cloud Services and Integrations (good to have):

• Manage containerized data applications on AWS ECS for efficient deployment and scaling.
• Design and implement event-driven architectures using AWS SQS and SNS.
• Configure AWS CloudFront for secure, low-latency content delivery.

• Security and Access Management (should understand how its implemented):

• Design and implement IAM roles and policies to ensure secure access and compliance.
• Ensure data security, including encryption at rest and in transit.

• Collaboration and Agile Practices:

• Collaborate with cross-functional teams, including data scientists, analysts, and developers, to understand data requirements.
• Actively participate in Agile ceremonies, such as sprint planning, standups, and retrospectives.

• Version Control and Continuous Integration:

• Utilize GitHub for version control, collaboration, and CI/CD workflows.
• Ensure best practices in code management and deployment processes.

• Monitoring and Optimization:

• Monitor and troubleshoot data workflows and AWS resources using AWS CloudWatch and other tools.
• Optimize data pipelines and resource configurations for performance and cost.