Data Engineer

Responsibilities:

• AI-Driven Automation: Design and implement AI systems that automate pipeline creation, data transformation logic, and workflow orchestration, pushing the boundaries of what's possible in data engineering.
• Pipeline Development: Build scalable, high-performance ETL pipelines that handle critical financial market data with reliability and efficiency.
• Event-Driven Architecture: Design event-driven ETL systems to support both real time and batch data processing needs Intelligent Infrastructure: Experiment with and deploy AI agents, code generation tools, and LLM-powered solutions to accelerate data platform development.
• Analytics and Insights: Run analytics on large datasets to uncover insights, validate models, and propose new platform features.
• Data Modelling: Architect data structures for both analytical (OLAP) and transactional systems using appropriate database technologies.
• Cross-Team Collaboration: Partner with engineers across Bangalore and New York to maintain data integrity, security, and performance standards.

Requirements:

• A bachelor's or master's degree in computer science or a comparable subject.
• 3+ years of software engineering experience with a focus on data engineering.
• Exceptional programming skills in Python and SQL.
• Expertise with data manipulation libraries (Pandas, Polars).
• Experience building production-grade ETL pipelines using orchestration tools (Airflow, Dagster, Prefect).
• Hands-on experience with DBT and DuckDB for data transformation and querying.
• Familiarity with event-based ETL architectures for data processing.
• Deep understanding of database technologies and when to use them; experience with Snowflake, MSSQL, Postgres, or similar.
• Passion for AI: Genuine curiosity about how AI can transform data engineering; experience using AI tools in development workflows (Claude, GitHub Copilot, or similar).
• Strong problem-solving abilities and comfort handling ambiguity.
• Excellent communication skills for collaborating across distributed, cross-functional teams.

Strongly Preferred:

• Experience building resilient data pipelines on Databricks using Delta Lake, Auto Loader, and workflow orchestration.
• AI Implementation Experience: Hands-on experience with AI agentic frameworks (Claude Code, LangGraph), LLM APIs, prompt engineering, or building AI-powered automation tools.
• Exposure to financial domain analytics or experience working with hedge funds/asset managers.
• Understanding of financial data models (positions, securities, market data, risk metrics).
• Experience with cloud platforms (Azure, AWS, GCP) and cloud-native data architectures.
• Familiarity with code generation, AI-assisted development, or building developer tools.
• Familiarity with data quality frameworks and monitoring systems.
• Strategic thinking with the ability to contribute to product roadmap discussions.
• Computer science, engineering, or related degree.