AWS Engineer

Lead the design, build, and operation of a secure, scalable migration platform to move object data from OBSv2 to AWS S3. The role focuses on reliable execution, automation, governance, monitoring, and production support for large-scale enterprise data migration.

Responsibilities:

• Own the end-to-end migration workflow from OBSv2 source buckets to AWS S3 destination buckets.
• Build and operate AWS-based orchestration using Step Functions, Lambda, Fargate, AWS Batch, DynamoDB, SQS, and rclone.
• Support both manifest-based and dynamic migration approaches, including batching, sharding, task sizing, and retry handling.
• Implement governance for bandwidth allocation, queue management, prioritization, throttling, and job tracking.
• Build observability through CloudWatch dashboards, logs, metrics, alerts, health checks, and progress reporting.
• Support secure multi-account deployment, IAM controls, privacy safeguards, testing, performance tuning, and production cutover.

Expected Deliverables:

• Production-ready migration orchestration platform supporting both manifest and no-manifest paths.
• Hub-side governance for bandwidth control, queue processing, throttling, and completion reporting.
• Reusable rclone-based migration runtime with progress reporting and dynamic throttle support.
• DynamoDB-backed job, task, queue, bandwidth, and summary state management.
• CloudWatch dashboards, alarms, logs, runbooks, deployment automation, and operational documentation.

Requirements:

• AWS Serverless: Hands-on experience with AWS Lambda, Step Functions, SQS, SNS, EventBridge, API Gateway, SSM Parameter Store, and Secrets Manager.
• AWS Compute and Containers: Strong working knowledge of ECS Fargate, AWS Batch, container task definitions, sidecar patterns, resource sizing, and runtime environment configuration.
• AWS: Storage; deep understanding of Amazon S3, S3 Intelligent-Tiering, S3-compatible APIs, object metadata, prefix structures, large object transfers, and lifecycle considerations.
• Object Storage Migration: Experience migrating large object datasets from on-premises or S3-compatible platforms such as Dell ECS/OBSv2 to AWS S3.
• rclone: Hands-on experience configuring and operating rclone for high-volume transfers, including transfers, checkers, retries, checksum validation, no-traverse behavior, stats, logs, and bandwidth limiting.
• Python Development: Strong Python skills for Lambda and Fargate services, including boto3, JSON processing, exception handling, logging, dataclasses, streaming files, and API integration.
• DynamoDB: Ability to design and operate DynamoDB tables, keys, indexes, scans, updates, batch writes, decimal handling, and state management patterns.
• Networking: Understanding of AWS Direct Connect, Transit Gateway, VPCs, private subnets, routing, S3 Gateway Endpoints, NAT Gateway scoping, and high-throughput network design.
• IAM and Security: Strong knowledge of IAM roles, cross-account access, STS assume-role, API keys, Secrets Manager, least-privilege design, and secure operational access.
• Observability: Experience with CloudWatch metrics, logs, alarms, dashboards, structured logging, operational telemetry, and migration progress reporting.
• CI/CD and Deployment: Experience with GitLab repositories, CI/CD pipelines, environment-specific configuration, infrastructure, deployment, and safe teardown procedures.

Preferred Experience:

• Experience with Dell ECS, OBSv2, or other S3-compatible enterprise object storage platforms.
• Experience delivering enterprise-scale migrations in regulated, production environments.
• Experience with hub-and-spoke operating models across multiple AWS accounts.
• Ability to produce clear technical documentation and leadership-ready reporting.

Know-How Skills and Domain Understanding:

• Understand large-scale object storage migration patterns, including manifests, prefix-based partitioning, batching, and validation.
• Balance migration performance across data volume, object count, compute resources, network capacity, and rclone concurrency.
• Design fair bandwidth allocation, throttling, queue handling, retry, and escalation models for production migrations.
• Apply strong operational discipline across monitoring, runbooks, restart procedures, completion checks, and stakeholder reporting.
• Maintain clear technical documentation, architecture notes, input contracts, implementation details, and support procedures.

Skills

AWS, AWS Lambda, Amazon S3, Amazon Web Services, Lambda, S3