Senior Data Engineer - Spark/Python

Role Overview :

As a Senior Data Engineer, you will serve as a cornerstone of our data infrastructure, architecting and maintaining robust pipelines that transform raw data into actionable business intelligence. You will work closely with cross-functional teams, including data scientists, product managers, and software engineers, to ensure high-quality data availability for critical decision-making.

By designing scalable systems and optimizing data flow, you will directly influence the efficiency of our analytics platforms and drive data-backed outcomes that enhance user experiences and operational performance across the organization. This role offers the flexibility to work from anywhere in India, allowing you to collaborate seamlessly with distributed teams in a high-impact environment.

Key Responsibilities :

- Architect and maintain high-performance data pipelines using Spark and Python to ensure seamless data ingestion and transformation for downstream analytics.

- Develop and optimize real-time data streaming solutions using Kafka to provide stakeholders with timely, accurate insights for critical business operations.

- Design and manage scalable data warehousing solutions within Snowflake DB to support complex analytical queries and improve overall data accessibility.

- Collaborate with engineering teams to migrate and manage data infrastructure on cloud platforms, ensuring high availability, security, and cost-efficiency.

- Implement best practices in code quality and system architecture using Scala to build resilient, maintainable data products that scale with business growth.

Required Skillset :

- Demonstrated expertise in building distributed data systems using Spark and Scala, with a strong command of Python for automation and data manipulation.

- Proven ability to design event-driven architectures using Kafka and manage large-scale data storage within Snowflake DB environments.

- Proficiency in deploying and managing cloud-native data solutions, ensuring robust performance and security across distributed environments.

- Strong interpersonal skills with the ability to communicate complex technical concepts to non-technical stakeholders and work effectively in a remote-first, collaborative setting.

- A proactive mindset with the adaptability to thrive in a fast-paced, evolving technical landscape, backed by a solid foundation in computer science or a related quantitative field.

- Ability to work independently and manage project timelines effectively while maintaining high standards of code quality and system reliability.

Skills

Data Engineering, Data Pipeline, Python, Spark, Kafka, Data Warehousing, Snowflake DB, Scala