Custom Software Engineer
Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs.
Must have skills : AWS Glue
Good to have skills : NA
Minimum 12 year(s) of experience is required
Educational Qualification : 15 years full time education
Summary
As a Senior AWS Glue Data Engineer / AWS Glue Architect, you will design, develop, and implement enterprise-scale ETL and data integration solutions using AWS Glue and AWS cloud-native data services. The role requires strong hands-on expertise in AWS Glue ETL development, PySpark programming, AWS Lambda, IAM security, VPC networking, and serverless data engineering architectures.
The engineer will build scalable data pipelines, metadata-driven frameworks, and cloud-native integration solutions using AWS Glue, Glue Data Catalog, Lambda, S3, Athena, Redshift, Step Functions, EventBridge, and other AWS services. The role involves data ingestion, transformation, performance optimization, security implementation, monitoring, and production support following AWS best practices.
Roles & Responsibilities
1. Design, develop, and maintain enterprise-scale ETL pipelines using AWS Glue.
2. Develop and optimize AWS Glue Spark jobs using PySpark for large-scale data processing.
3. Create and manage AWS Glue components including Jobs, Crawlers, Data Catalog, Workflows, Triggers, Connections, and Job Bookmarks.
4. Build metadata-driven ETL frameworks using AWS Glue Data Catalog and configuration-based processing approaches.
5. Develop data ingestion and transformation pipelines from multiple sources including databases, APIs, files, and cloud platforms.
6. Integrate AWS Glue with AWS services including Lambda, S3, Athena, Redshift, Step Functions, EventBridge, and SQS.
7. Develop AWS Lambda functions for data processing, automation, event-driven workflows, and Glue job orchestration.
8. Configure and manage AWS IAM Roles, Policies, Permissions, and Security Controls for Glue jobs and AWS services.
9. Design and configure AWS VPC components including VPC endpoints, subnets, security groups, route tables, and network connectivity for AWS Glue workloads.
10. Configure AWS Glue connections for databases using JDBC connectivity.
11. Implement secure data integration solutions using JDBC, REST APIs, SFTP, and event-driven architectures.
12. Perform AWS Glue job tuning using Spark optimization techniques, partitioning, compression, caching, and performance best practices.
13. Implement data quality checks, validation frameworks, error handling, and reconciliation processes.
14. Monitor and troubleshoot AWS Glue jobs using Amazon CloudWatch, CloudWatch Logs, and AWS monitoring services.
15. Implement CI/CD pipelines for AWS Glue deployments using Git, Jenkins, GitHub Actions, AWS CodePipeline, or AWS CodeBuild.
16. Develop Infrastructure as Code (IaC) solutions using AWS CloudFormation or AWS CDK.
17. Support production incidents, troubleshooting, performance issues, and operational improvements.
18. Collaborate with data architects, data engineers, application teams, and business stakeholders to deliver AWS data solutions.
19. Mentor team members on AWS Glue development standards, PySpark optimization, and AWS data engineering best practices.
Professional & Technical Skills
1. Strong hands-on experience with AWS Glue for ETL development, data integration, and data pipeline implementation.
2. Strong programming skills in Python, PySpark, and SQL for data transformation and processing.
3. Experience with AWS services including Lambda, S3, Athena, Redshift, Step Functions, and EventBridge.
4. Strong knowledge of AWS IAM Roles, Policies, Security Configuration, and VPC setup for secure data solutions.
5. Experience designing and optimizing AWS Glue ETL pipelines, Spark jobs, and data processing workflows.
6. Good understanding of Data Lake architecture, AWS Data Catalog, metadata management, and data governance.
7. Experience with CI/CD, Git, Infrastructure as Code (CloudFormation/Terraform), Agile, and DevOps practices.
Additional Information
1. The candidate should have a minimum of 10 years of experience in AWS Glue.
2. This position is based at the Hyderabad office.
3. A minimum of 15 years of full-time education is required.
15 years full time educationAbout Accenture
Accenture is a leading global professional services company that helps the world’s leading businesses, governments and other organizations build their digital core, optimize their operations, accelerate revenue growth and enhance citizen services-creating tangible value at speed and scale. We are a talent- and innovation-led company with approximately 791,000 people serving clients in more than 120 countries. Technology is at the core of change today, and we are one of the world’s leaders in helping drive that change, with strong ecosystem relationships. We combine our strength in technology and leadership in cloud, data and AI with unmatched industry experience, functional expertise and global delivery capability. Our broad range of services, solutions and assets across Strategy & Consulting, Technology, Operations, Industry X and Song, together with our culture of shared success and commitment to creating 360° value, enable us to help our clients reinvent and build trusted, lasting relationships. We measure our success by the 360° value we create for our clients, each other, our shareholders, partners and communities.Visit us atwww.accenture.com
Equal Employment Opportunity Statement
We believe that no one should be discriminated against because of their differences. All employment decisions shall be made without regard to age, race, creed, color, religion, sex, national origin, ancestry, disability status, militaryveteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by applicablelaw. Our rich diversity makes us more innovative, more competitive, and more creative, which helps us better serve our clients and our communities.