Senior Data Engineer

Responsibilities:

• Design, build, and manage scalable data processing pipelines on the Google Cloud Platform.
• Drive engineering excellence, data quality, and continuous improvement.
• Lead end-to-end design, development, and deployment of scalable data pipelines and integration solutions.
• Manage multiple delivery workstreams.
• Translate complex business and technical requirements into robust data engineering solutions.
• Develop and optimise ETL/ELT processes using modern cloud, data warehousing, and data processing platforms.
• Collaborate with technical and business teams to translate complex requirements into robust data solutions.
• Diagnose and resolve complex production issues through root-cause analysis and system optimisation.
• Drive technology and process improvements by evaluating emerging tools and engineering best practices.
• Ensure adherence to enterprise architecture, security, compliance, and performance standards.
• Maintain high-quality technical documentation, user guides, and solution specifications.

Requirements:

• Experience in data modelling, data engineering, and data integration.
• Experience leading engineering initiatives.
• Proven experience with cloud tools such as GCP Dataflow, GCP Composer (Airflow), and GCP Cloud Storage.
• Strong expertise in cloud platforms: GCP.
• Hands-on experience with data warehousing and processing technologies.
• Experience with data streaming tools: Kafka, Cloud Data Flow, Spark Streaming.
• Strong experience in data quality and observability platforms to automate quality coverage using iceDQ, Monte Carlo, or similar tools.
• Strong proficiency with high-level programming and query languages.
• Experience collaborating with distributed teams across time zones.
• Skills: BigQuery, Lakehouse (Iceberg), Spark, Python, SQL.

AI Skills (Mandatory):

• Proficiency in leveraging AI tools for daily engineering tasks to enhance productivity, optimise effort, and ensure cost-aware AI assistance.
• Familiarity with Retrieval Augmented Generation (RAG) architectures, including semantic layers for data retrieval and grounding Large Language Model (LLM) responses.
• Conceptual understanding of LLM-based applications (e. g., chatbots, Q& A systems), encompassing prompt engineering, context management, and response generation.
• Exposure to agentic and multi-agent AI systems, including agent roles, tool utilisation, memory management, and workflow orchestration.
• Practical experience with LLM application frameworks (e. g., LangChain, Lang Graph, or Google GenAI tools) for prototyping and integrating AI-driven solutions.

Nice to Have:

• Experience with Power BI/ThoughtSpot, AI in data engineering, Stonebranch, CI/CD, Databricks and DevOps practices.
• Familiarity with enterprise databases such as DB2 SQL Server, Oracle, and PostgreSQL.
• Google Cloud Data Engineer certification is preferred.
• Experience in retail domains (customer, stores, eCommerce, marketing, supply chain, and merchandising).

Skills

Airflow, BigQuery, Dataflow, ELT, ETL, GCP, Google Cloud Platform, Iceberg, Kafka, Python, SQL, Spark, data engineering, data integration, data modeling