Data Engineer

We are looking for a skilled Data Engineer with 3-4 years of experience in developing, maintaining, and optimizing data pipelines and data processing solutions. The ideal candidate should have strong hands-on experience in Python, SQL, PySpark, and big data technologies, along with exposure to cloud-based data platforms.

Responsibilities:

• Design, develop, and maintain scalable data pipelines for processing structured and unstructured data.
• Develop data transformation and processing workflows using Python, PySpark, and SQL.
• Work with large datasets using Apache Spark and related big data technologies.
• Build and maintain ETL/ELT pipelines to extract, transform, and load data from multiple sources.
• Work with data storage and processing technologies such as Hive, Hadoop, and cloud-based data platforms.
• Optimize SQL queries and data processing jobs for improved performance.
• Perform data validation, troubleshoot pipeline failures, and ensure data quality.
• Collaborate with data analysts, data scientists, and other engineering teams to support business requirements.
• Monitor scheduled jobs and ensure the reliability and efficiency of data pipelines.

Requirements:

• 3-4 years of relevant experience in data engineering.
• Strong problem-solving and analytical skills.
• Ability to work independently on data engineering tasks.
• Good communication and collaboration skills.

Required Skills:

• Python: Strong hands-on experience in Python programming and data processing.
• SQL: Good knowledge of complex SQL queries, joins, aggregations, and query optimization.
• PySpark / Apache Spark: Experience in distributed data processing and large-scale data transformations.
• ETL/ELT: Hands-on experience building and maintaining data pipelines.
• Big Data Technologies: Exposure to Hive, Hadoop, Spark SQL, or similar technologies.
• Data Processing: Understanding of data warehousing concepts, data modeling, and file formats such as Parquet and JSON.
• Workflow Orchestration: Exposure to Apache Airflow or similar scheduling tools.
• Cloud Platforms: Exposure to Azure, AWS, or GCP is preferred.
• Version Control: Working knowledge of Git.

Good to Have:

• Experience with Azure Databricks, Azure Data Factory, or similar cloud data engineering services.
• Familiarity with data lake and data warehouse architectures.
• Experience with performance tuning and optimizing Spark jobs.
• Understanding of data quality checks and pipeline monitoring.

Skills

Apache Spark, PySpark, Python, SQL, Spark