Senior Specialist - Data Engineering

Role: PySpark Application Support Lead

Experience: 8-12 Years

Location :Anywhere in India

Focus Production Support Troubleshooting Optimization Team Leadership

Role Summary

Leadership Skills and proven experience in leading technical teams handling escalations and conducting knowledgesharing sessions The Pyspark Application support is responsible for the monitoring and troubleshooting Pyspark Jobs on the Cloudera environment This role ensures high availability performance and security of ETL services while supporting developers analysts and business users across environments Prod and Nonprod

Technical Skills Primary

PySpark Application support

Handson experience with Pyspark jobs development ETL support

Monitor PySpark batch jobs streaming pipelines and scheduled workflows Airflow ADF Oozie etcChecking for failures and performance degradation

Troubleshoot job failures performance degradation schema mismatches data quality issues

Resolve errors related to Spark SQL DataFrame transformations cluster configuration and storage layers

Handle failures caused by upstreamdownstream systems Kafka Delta Lake Hive S3ADLS etc

Coordinating with development team for fixing the failures

Coordinating with QA team on data validation failures

Supporting in production release and change management process

Incident Problem Management

Root cause analysis for recurring issues

Performance tuning at a basic level cache queues agents

Familiarity with ITIL processes for AMS support

Tools

Ticketing systems ServiceNowJira

Monitoring tools

Version control basics GitSVN for migration scripts

Soft Skills

Ability to work with client stakeholders

Good communication with analytical and problemsolving skills

Ability to Effectively interact with stakeholders colleagues and other members of the team

Demonstrated ability to be organized and selfdirected

Strong interpersonal skills with the ability to work cross functionally

Responsibilities

Monitoring Maintenance

Proactively monitor Informatica environments for uptime and performance

Execute scheduled health checks and housekeeping tasks

Maintain logs and escalate anomalies promptly

Incident Management

Respond to incidents within SLA timelines

Perform initial triage resolve L2 issues and escalate L3 as needed

Document RCA and preventive measures

User Support

Assist users with access issues report failures and portal navigation

Manage user provisioning and security roles

Change Release Support

Support deployments and migrations between environments

Validate postdeployment checks and rollback plans

Compliance Documentation

Ensure adherence to security and audit requirements

Maintain SOPs runbooks and knowledge base for AMS operations

Production Support Incident Management Act as the primary point of contact for highpriority incidents debugging and resolving complex PySpark application issues in production

Technical Leadership Lead the efforts to design build and configure improvements to existing data pipelines and applications

Team Management Mentorship Guide mentor and manage a team of support specialistsdevelopers to achieve operational excellence and meet project deadlines

Performance Optimization Analyze existing PySpark applications and data pipelines for performance bottlenecks and implement optimization techniques eg Spark UI analysis tuning executors

Stakeholder Interaction Collaborate with crossfunctional teams developers Data Engineers DevOps to ensure seamless application functionality

Documentation Best Practices Create maintain and update technical documentation standard operating procedures SOPs and ensure code compliance with best practices

Automation Implement monitoring improvements and automate recurring manual tasks