Data Engineering (Python +PySpark)

Role: Developer. Skills: Big Data, Python. Experience: 4 - 10 Years. Qualification: Bachelor Of Engineering

Desired Competencies

(Technical/Behavioral Competency)

Must-Have

Strong

hands-on experience in Python and PySpark for data processing and data

engineering activities.

Experience in developing data solutions using PySpark, Spark SQL, and

related frameworks/libraries.

Hands-on experience in building and maintaining ETL/ELT pipelines, data

ingestion pipelines, and data transformation processes.

Experience in ingesting data from multiple sources such as databases,

files, cloud storage, APIs, S3/data lake platforms, or similar.

Experience working with structured and unstructured data.

Good understanding of data warehouse concepts, data lake concepts, and

data processing patterns.

Ability to develop scalable, reusable, and maintainable data processing

components.

Experience in end-to-end data pipeline development, including source

ingestion, transformation, validation, and target load.

Good knowledge of SQL for data analysis, transformation, validation,

and basic performance tuning.

Ability to write clean, efficient, reusable, and scalable Python code.

Good understanding of data quality checks, testing, monitoring,

documentation, and production support practices.

Awareness of security and data protection principles in data

engineering solutions.

Good-to-Have

Exposure to cloud platforms such as AWS,

Azure, or GCP and related data services.

Experience with cloud storage/data platforms such as S3, ADLS, Blob

Storage, Databricks, EMR, Synapse, or similar.

Knowledge of orchestration/scheduling tools such as Airflow, Control-M,

Azure Data Factory, AWS Glue, Oozie, or similar.

Exposure to CI/CD, automation, pipeline deployment, and monitoring

concepts.

Knowledge of data management principles, metadata management, data

governance, and data lineage.

Experience with Python frameworks such as Flask, Django, or FastAPI

will be an added advantage.

Knowledge of ORM concepts will be preferred where application

integration is required.

Exposure to streaming data processing using Kafka, Spark Streaming,

Kinesis, Event Hubs, or similar.

Ability to create and maintain technical documents, data mapping documents,

and support documentation.