Senior Data Engineer

Senior Data Engineer

About the Role :

We are looking for an experienced Senior Data Engineer with strong expertise in Azure Databricks, data engineering, and scalable data pipelines.

The successful candidate will be responsible for designing, developing, testing, implementing, and optimizing data pipelines and data processing solutions. The role requires strong technical expertise, attention to data quality, and the ability to collaborate with architects, analysts, data scientists, business stakeholders, and technology teams.

Job Details :

Job Title : Senior Data Engineer

Department & Team : Technology

Reporting To : Big Data Manager

Work Mode : Remote

Office Location : Any Wipro Location Pan India

Experience : 8+ Years

Notice Period : Immediate Joiners Only

Role Purpose :

As part of the ongoing Data Transformation initiative, data is fundamental to business operations. The objective of this role is to ensure that the right data is available in the right place at the right time and is accurate, consistent, secure, and reliable.

The Senior Data Engineer will be responsible for designing and implementing scalable data pipelines, ensuring data quality and reliability, and supporting the development of production-ready data solutions.

Key Responsibilities :

- Design, develop, and implement scalable data pipelines to collect, clean, transform, and process data from multiple sources.

- Build and maintain data storage and processing systems, including databases, data warehouses, and data lakes.

- Work extensively with Azure Databricks to develop and optimize data engineering solutions.

- Develop data processing applications using Python and PySpark.

- Collaborate with Big Data Solution Architects to design, prototype, implement, and optimize data ingestion pipelines.

- Ensure data is shared effectively across various business systems.

- Ensure solutions are production-ready from operational, security, compliance, performance, and reliability perspectives.

- Implement appropriate data security and protection measures.

- Contribute to data governance policies, standards, and procedures.

- Collaborate with Data Analysts, Data Scientists, Business Analysts, Architects, and other technology teams to understand data requirements and deliver appropriate solutions.

- Develop effective and maintainable unit and integration tests for data ingestion pipelines.

- Identify and resolve data quality and pipeline-related issues.

- Participate in Agile ceremonies, sprint activities, technical discussions, and project meetings.

- Provide technical support and contribute to faster resolution of production and development issues.

- Clearly communicate project status, technical updates, risks, dependencies, and blockers to relevant stakeholders.

- Share technical knowledge with wider business and technology teams and contribute to documentation of processes and ways of working.

- Maintain an end-to-end understanding of the data landscape and dependencies across business systems.

- Work on data pipeline performance, reliability, resiliency, and processing efficiency.

- Support CI/CD implementation and use appropriate static analysis and code quality tools.

Required Technical Skills :

Data Engineering :

- 8+ years of experience in Data Engineering / Big Data development.

- Strong experience designing and developing data pipelines.

- Strong hands-on experience with Azure Databricks.

- Strong experience with Python and PySpark.

- Experience working with large-scale data processing environments.

- Experience with data ingestion, transformation, and processing pipelines.

Cloud Technologies :

- 2+ years of hands-on development experience with cloud technologies such as Azure, AWS, or GCP.

- Strong preference for candidates with extensive Azure experience.

- Experience with cloud-based data engineering architectures and services.

Databases & Data Platforms :

- Proficient in querying and manipulating data from relational databases and Big Data platforms.

- Experience working with data warehouses, data lakes, and large-scale data processing systems.

DevOps & Infrastructure :

- Experience with Kubernetes.

- Experience with Terraform.

- Experience building and maintaining CI/CD pipelines.

- Experience using static analysis and code quality tools.

Testing & Quality :

- Experience writing effective and maintainable unit tests.

- Experience developing integration tests for data ingestion pipelines.

- Strong understanding of data quality, reliability, and resiliency.

- Experience with defect analysis and root-cause investigation.

Large-Scale Systems :

- Experience working on high-traffic and large-scale software products.

- Strong understanding of performance, scalability, reliability, and production readiness.

Behavioural Skills :

- Strong technical aptitude with a keen eye for detail.

- Self-driven, self-motivated, and results-oriented.

- Confident in challenging existing processes or solutions when required.

- Process-driven, organized, and well-structured.

- Excellent communication and stakeholder management skills.

- Strong problem-solving and analytical abilities.

- Ability to work effectively in a cross-functional and multicultural environment.

- Ability to work collaboratively with minimal supervision.

- Ability to multitask and manage multiple priorities effectively.

- Comfortable working in a fast-paced and dynamic environment.

Key Performance Indicators :

- Data pipeline reliability and resiliency.

- Data processing efficiency.

- Release quality.

- Defect analysis and resolution.

- Data quality maintained throughout development and deployment.

- Time to market.

- Resource utilization.

- Business and stakeholder satisfaction.

Skills

Data Engineering, Azure Databricks, Data Pipeline, Python, PySpark, Data Warehousing, AWS, Data Ingestion