Senior Specialist - Data Engineering
Confirmed on the employer's own hiring board on Oct 18, 2026. First seen by Alion on Sep 24, 2026. Ltm scores B on the Alion truth index.
Role: PySpark Application Support Lead
Experience: 8-12 Years
Location :Anywhere in India
Focus Production Support Troubleshooting Optimization Team Leadership
Role Summary
Leadership Skills and proven experience in leading technical teams handling escalations and conducting knowledgesharing sessions The Pyspark Application support is responsible for the monitoring and troubleshooting Pyspark Jobs on the Cloudera environment This role ensures high availability performance and security of ETL services while supporting developers analysts and business users across environments Prod and Nonprod
Technical Skills Primary
PySpark Application support
Handson experience with Pyspark jobs development ETL support
Monitor PySpark batch jobs streaming pipelines and scheduled workflows Airflow ADF Oozie etcChecking for failures and performance degradation
Troubleshoot job failures performance degradation schema mismatches data quality issues
Resolve errors related to Spark SQL DataFrame transformations cluster configuration and storage layers
Handle failures caused by upstreamdownstream systems Kafka Delta Lake Hive S3ADLS etc
Coordinating with development team for fixing the failures
Coordinating with QA team on data validation failures
Supporting in production release and change management process
Incident Problem Management
Root cause analysis for recurring issues
Performance tuning at a basic level cache queues agents
Familiarity with ITIL processes for AMS support
Tools
Ticketing systems ServiceNowJira
Monitoring tools
Version control basics GitSVN for migration scripts
Soft Skills
Ability to work with client stakeholders
Good communication with analytical and problemsolving skills
Ability to Effectively interact with stakeholders colleagues and other members of the team
Demonstrated ability to be organized and selfdirected
Strong interpersonal skills with the ability to work cross functionally
Responsibilities
Monitoring Maintenance
Proactively monitor Informatica environments for uptime and performance
Execute scheduled health checks and housekeeping tasks
Maintain logs and escalate anomalies promptly
Incident Management
Respond to incidents within SLA timelines
Perform initial triage resolve L2 issues and escalate L3 as needed
Document RCA and preventive measures
User Support
Assist users with access issues report failures and portal navigation
Manage user provisioning and security roles
Change Release Support
Support deployments and migrations between environments
Validate postdeployment checks and rollback plans
Compliance Documentation
Ensure adherence to security and audit requirements
Maintain SOPs runbooks and knowledge base for AMS operations
Production Support Incident Management Act as the primary point of contact for highpriority incidents debugging and resolving complex PySpark application issues in production
Technical Leadership Lead the efforts to design build and configure improvements to existing data pipelines and applications
Team Management Mentorship Guide mentor and manage a team of support specialistsdevelopers to achieve operational excellence and meet project deadlines
Performance Optimization Analyze existing PySpark applications and data pipelines for performance bottlenecks and implement optimization techniques eg Spark UI analysis tuning executors
Stakeholder Interaction Collaborate with crossfunctional teams developers Data Engineers DevOps to ensure seamless application functionality
Documentation Best Practices Create maintain and update technical documentation standard operating procedures SOPs and ensure code compliance with best practices
Automation Implement monitoring improvements and automate recurring manual tasks