Lead Data Engineer

Job Title: Lead Data Engineer

Location: Charlotte, NC

Duration: 12 months

Pay Rate: $90 - $100/HR (W2 Only)

Job/Role Description:

• Lead the design, architecture, and implementation of enterprise-scale data engineering solutions across AWS cloud environments.
• Design, develop, and optimize scalable, resilient data pipelines, ETL processes, data ingestion frameworks, and orchestration workflows.
• Architect and oversee enterprise data lake and data warehouse solutions using AWS Lake Formation, Amazon Redshift, Amazon Athena, S3, and related AWS technologies.
• Collaborate with Lead Developers, Data Scientists, Architects, Product Owners, and business stakeholders to define technical strategy and scalable data solutions.
• Provide hands-on technical leadership, engineering oversight, code reviews, and mentorship to Data Engineers and development teams.
• Drive architectural decisions in partnership with Data and Solution Architects to ensure scalability, security, reliability, performance, and maintainability.
• Design and support Kafka-based streaming and event-driven data architectures, preferably using Confluent Kafka.
• Develop distributed data processing solutions using Python, PySpark, AWS EMR, and other cloud-native technologies.
• Establish engineering standards and best practices for data modeling, ETL frameworks, pipeline reliability, monitoring, observability, and operational excellence.
• Lead end-to-end solution delivery while ensuring alignment with business requirements, enterprise architecture standards, security controls, and regulatory requirements.
• Design and implement Infrastructure as Code solutions using Terraform across multiple AWS accounts and environments.
• Develop and maintain CI/CD frameworks using GitHub and GitHub Actions.
• Oversee production support and operational management of AWS-based data platforms, including root-cause analysis, troubleshooting, and performance optimization.
• Champion data governance, metadata management, data quality, observability, and data stewardship practices across platforms and teams.
• Identify opportunities to modernize data architecture and improve operational efficiency through automation and cloud-native technologies.
• Support the development of AI-ready data pipelines and machine learning workflows, including feature engineering and MLOps.
• Design data solutions capable of supporting Generative AI applications, intelligent search, RAG architectures, LLMs, and vector databases.
• Drive technical decision-making and clearly communicate complex data architecture concepts to both technical and non-technical stakeholders.

Required Qualifications

• 8+ years of Data Engineering experience, including at least 5+ years of extensive experience working within AWS environments.
• Expert-level experience with AWS services including S3, EMR, Glue Jobs, Lambda, Athena, CloudTrail, SNS, SQS, CloudWatch, and Step Functions.
• Strong hands-on experience designing and implementing enterprise-scale data lakes and data warehouses using AWS Lake Formation, Amazon Redshift, and Amazon Athena.
• Advanced Python development experience with extensive hands-on use of PySpark.
• Advanced SQL expertise, including query optimization, large-scale data processing, and enterprise data warehousing.
• Strong data modeling experience, including dimensional modeling, Data Vault, and other enterprise data modeling techniques.
• Deep experience designing, developing, and optimizing scalable and resilient data pipelines within AWS environments.
• Strong experience with distributed data processing frameworks, particularly PySpark and AWS EMR.
• Extensive knowledge of database management, performance tuning, and data architecture best practices.
• Experience with Kafka-based streaming architectures, preferably Confluent Kafka.
• Expertise with Infrastructure as Code using Terraform.
• Experience designing and implementing CI/CD frameworks using GitHub and GitHub Actions.
• Deep knowledge of AWS IAM roles, policies, governance, security controls, and cloud security best practices.
• Strong experience with workflow orchestration platforms such as AWS Step Functions, Apache Airflow, or equivalent technologies.
• Experience leading cloud migration, modernization, or enterprise data platform initiatives.
• Strong understanding of data governance, metadata management, data quality frameworks, observability, resiliency, and operational supportability.
• Experience creating AI applications using AWS Bedrock.
• Experience building AI-ready data pipelines and ML workflows, including feature engineering and MLOps.
• Knowledge of Generative AI technologies, LLMs, Retrieval-Augmented Generation (RAG), and vector databases.
• Experience working with cloud-based AI and data platforms such as AWS, Azure, or Databricks.
• Ability to lead hands-on development efforts while providing technical direction, engineering oversight, and code reviews across multiple initiatives.
• Experience establishing cloud development environments, infrastructure standards, security controls, and migration strategies across multiple accounts and environments.
• Proven ability to identify data gaps, develop strategic remediation plans, and implement scalable automation solutions.
• Experience designing highly reliable data pipelines with a strong emphasis on data quality, observability, resiliency, and operational support