Mid-level Data Engineer

• Masters in computer applications or equivalent OR bachelors degree in engineering or computer science or equivalent.
• Deep understanding of Hadoop and Spark Architecture and its working principle.
• Deep understanding of Data warehousing concepts.
• Ability to design and develop optimized Data pipelines for batch and real time data processing.
• Experience in data analytics and cleansing
• 5+ years of software development experience.
• 5+ years experience on Python or JavaHands-on experience on writing and understanding complex SQL (Hive/PySpark-data frames), optimizing joins while processing huge amount of data.
• 3+ years of hands-on experience of working with Map-Reduce, Hive, Spark (core, SQLand PySpark).
• Hands on Experience on Google Cloud Platform (BigQuery, DataProc, Cloud Composer)
• Hands on Experience in Airflow
• 3+ years of experience in UNIX shell scripting
• Should have experience in analysis, design, development, testing, and implementation of system applications.
• Ability to effectively communicate with internal and external business partners.

• Understanding of Distributed eco system.
• Experience in machine learning models, RAG, and NLP
• Experience in designing and building solutions using Kafka streams or queues.
• Experience with NoSQL i.e., HBase, Cassandra, Couchbase or MongoDB
• Experience with Data Visualization tools like PowerBI, Tableau, SiSense, Looker
• Ability to learn and apply new programming concepts.
• Knowledge of Financial reporting ecosystem will be a plus.
• Experience in leading teams of engineers and scrum teams

Originally posted on Himalayas