Senior Data Engineer

Job Title: Sr. DATA Engineer

Work Location : Tampa FL

Job Description:

• Seeking a Senior professional with 3 to 5 years of experience in Python Scala and Spark SQL to design and optimize scalable big data processing solutions within the Databricks Spark SQL Scala Python Java ecosystem
• Develop and maintain scalable data processing pipelines using Apache Spark with Python and Scala
• Utilize Spark SQL to analyze large datasets and extract meaningful insights supporting data driven decision making
• Collaborate with cross functional teams to integrate big data solutions into existing systems
• Optimize Spark jobs for performance and efficiency in distributed computing environments
• Ensure data quality and consistency across various data sources
• Stay updated with the latest advancements in Spark Scala Python and related big data technologies to continuously improve processing capabilitiesÂ

  Roles and ResponsibilitiesÂ

  • Design develop and deploy complex data processing workflows leveraging Apache Spark Scala Python and SparkSQL
  • Write efficient reusable and maintainable code for Spark applications
  • Troubleshoot and resolve performance bottlenecks in Spark jobs and data pipelines
  • Collaborate with data engineers’ data scientists and stakeholders to translate business requirements into technical solutions
  • Conduct code reviews and provide mentorship to junior team members on best practices and coding standards
  • Participate in architectural discussions and contribute to the evolution of big data platforms
  • Monitor and maintain the health availability and reliability of Spark clusters and data services Karat Interview mandate