Data and AI Engineer

This role sits at the intersection of legacy big data infrastructure and a modern AI-ready cloud ecosystem. You will keep existing data pipelines reliable while re-architecting them onto AWS and Databricks. You will also build and deploy intelligent AI agents and LLM-powered workflows that deliver immediate business value.

Our client is a leading global technology and analytics company operating in the computer software industry. They serve large enterprises worldwide with data-driven solutions and are investing heavily in modern cloud platforms and generative AI capabilities. The organization is known for a collaborative, innovation-led culture that values engineering excellence.

Responsibilities

• Re-architect legacy Apache Spark and Hadoop MapReduce workflows into optimized, production-ready pipelines on AWS and Databricks.
• Design, build, and deploy AI agents and workflow automation tools using leading Large Language Models such as Anthropic Claude.
• Build AI-oriented data pipelines covering unstructured data ingestion, feature tokenization, and metadata tagging for vector databases.
• Monitor, maintain, and troubleshoot existing Hadoop big data workloads, resolving bottlenecks and out-of-memory errors during migration.
• Act as the engineering liaison for BI stakeholders, resolving dashboard latency and optimizing data extracts for tools such as Tableau, Looker, and Power BI.
• Use Databricks and Delta Lake features to improve pipeline performance, reliability, and cost efficiency.
• Leverage Databricks AI tools such as Vector Search, Mosaic AI, and Lakeflow to deliver production-grade Generative AI and RAG applications.

Requirements

• 5 to 8 years of professional data engineering or software engineering experience in production environments.
• Hands-on experience building AI agents, integrating LLM APIs (especially Anthropic Claude), and using orchestration frameworks such as LangChain or Databricks Mosaic AI.
• Solid hands-on experience with Apache Spark and a foundational understanding of partitioning, shuffling, caching, and broadcast joins.
• Strong proficiency in Python (PySpark) or Scala and intermediate-to-advanced SQL, including window functions, query tuning, and complex joins.
• Practical exposure to the AWS ecosystem (S3, IAM) and Databricks environments.
• Experience with at least one BI tool such as Tableau, Looker, or Power BI, including debugging calculated fields and troubleshooting slow reports.
• Bachelor's degree in Computer Science, Data Engineering, Information Systems, or a related quantitative field, or equivalent practical experience.

Skills

Apache Spark, PySpark, Python, SQL, AWS (S3, IAM), Databricks, Hadoop (HDFS, MapReduce, Hive), LLM integration (Anthropic Claude), LangChain, Scala, Airflow, Control-M, Delta Lake, Unity Catalog, Vector Search, Mosaic AI, AWS Glue, AWS EMR, Tableau, Looker, Power BI, Copilot, Lakeflow