Data Analytics and AI Engineer

This role bridges a mature big data environment and a future AI-ready cloud ecosystem. You will maintain and optimize existing data pipelines while re-architecting workflows onto AWS and Databricks. You will also contribute to building intelligent AI agents and LLM-powered applications that deliver immediate business value.

Our client is a leading global software company in the computer software industry, serving enterprises worldwide with data, analytics and technology solutions. They operate at significant scale with a strong focus on innovation and modern cloud platforms. The culture emphasizes continuous learning, engineering excellence and delivering measurable business value.

Responsibilities

• Re-architect legacy Apache Spark and Hadoop MapReduce workflows into optimized, production-ready pipelines on AWS and Databricks.
• Design, build and deploy AI agents and workflow automation tools leveraging Large Language Models such as Anthropic Claude.
• Build and optimize pipelines for AI use cases, including unstructured data ingestion, feature tokenization and metadata tagging for vector databases.
• Monitor, maintain and troubleshoot big data workloads on Hadoop clusters, resolving bottlenecks and out-of-memory errors during migration.
• Serve as engineering liaison for BI stakeholders, performing enhancements, bug fixes and data extract optimizations to reduce dashboard latency.
• Use Databricks and Delta Lake features such as ACID transactions, Z-Ordering and caching to improve pipeline performance, reliability and cost efficiency.
• Leverage Databricks Vector Search, Mosaic AI and Lakeflow to orchestrate and serve production-grade Generative AI and RAG applications.

Requirements

• 5 to 8 years of professional software or data engineering experience in production environments.
• Hands-on experience building AI agents, integrating LLM APIs (notably Anthropic Claude) and using orchestration frameworks such as LangChain or Databricks Mosaic AI.
• Solid hands-on experience with Apache Spark, including understanding of partitioning, shuffling, caching and broadcast joins.
• Strong proficiency in Python (PySpark) or Scala and intermediate-to-advanced SQL (window functions, query tuning, complex joins).
• Direct experience or deep knowledge of the AWS ecosystem (S3, IAM) and Databricks environments.
• Practical experience with any BI tool such as Tableau, Looker or Power BI, including debugging calculated fields and troubleshooting slow reports.
• Bachelor's degree in Computer Science, Data Engineering, Information Systems or a related quantitative field, or equivalent practical experience.

Skills

Apache Spark, Hadoop (HDFS, MapReduce, Hive), Python (PySpark) or Scala, SQL, AWS (S3, IAM), Databricks (Delta Lake), LLM APIs (Anthropic Claude), LangChain / AI agent orchestration frameworks, Tableau or other BI tools, AWS Glue, AWS EMR, Airflow, Control-M, Unity Catalog, Databricks Vector Search, Databricks Mosaic AI, Lakeflow, Copilot, RAG applications, Looker, Power BI