Platform Data Engineer
First seen by Alion on Sep 23, 2026.
Platform Data Engineer
Primary Skills :
- Azure Data Factory
- Databricks
- PySpark
- Azure Cloud Services
- Data Engineering, Kafka, Infrastructure Automation
Role Summary :
We are seeking an experienced Platform Data Engineer to join the Global Cybersecurity Data Acquisition and Engineering team. The role is responsible for designing, developing, and supporting enterprise-scale data ingestion pipelines that onboard security data into a cyber data lake and analytics platform.
The ideal candidate :
The ideal candidate will have strong expertise in Azure-based data engineering solutions, Databricks, PySpark, and cloud-native technologies. This role requires hands-on experience building reliable, scalable, and performant data pipelines while collaborating with cybersecurity, infrastructure, and analytics teams to deliver high-quality security data for advanced analytics and threat detection.
Key Responsibilities :
- Design, develop, and maintain scalable data ingestion and integration pipelines using Azure Data Factory, Databricks, and PySpark.
- Develop and support secure, reliable, and high-performance data acquisition patterns for the cyber analytics platform.
- Lead technical enhancements across existing and future analytics technologies including Databricks, Data Factory, Kafka, NiFi, and related platforms.
- Build and optimize automation and orchestration capabilities across the cybersecurity data platform.
- Support the development and deployment of security analytics and detection use cases.
- Collaborate with cybersecurity teams, infrastructure teams, and business stakeholders to implement data-driven security solutions.
- Drive continuous improvement initiatives across platform engineering, data quality, monitoring, and operational processes.
- Provide technical guidance and mentorship to team members and promote engineering best practices.
- Support platform monitoring, troubleshooting, and performance optimization activities.
- Ensure solutions follow security, governance, and operational excellence standards.
Primary Skills (Must Have) :
1. Data Engineering & Analytics :
- Azure Data Factory (ADF)
- Databricks
- PySpark
- Data Pipeline Development
- Data Ingestion & Transformation
- ETL / ELT Processes
- Data Quality and Data Cleansing
2. Cloud Platform :
- Microsoft Azure
- Azure Storage Services
- Azure Functions
- Azure Compute Services
- Azure Networking
- Azure Identity & Access Management
3. Streaming & Real-Time Processing :
- Apache Kafka
- Azure Event Hubs
- Spark Streaming
- Real-Time Data Processing
4. Infrastructure & Automation :
- Infrastructure as Code (Terraform / ARM)
- Automation Tools
- CI/CD Concepts
- Cloud Infrastructure Management
5. Operating Systems :
- Linux Administration (RHEL / Ubuntu)
Secondary Skills (Nice to Have) :
- Apache NiFi
- Fluentd
- Azure Power BI
- Azure Containers and Kubernetes
- Chef / Ansible
- Cloud-based SIEM Platforms
- SOAR Technologies
- Cybersecurity Analytics Platforms
- Security Data Management
Key Competencies :
- Strong leadership and technical problem-solving abilities.
- Excellent stakeholder management and collaboration skills.
- Experience working in large-scale enterprise environments.
- Ability to design sustainable, scalable, and secure platform solutions.
- Strong troubleshooting and performance optimization expertise.
- Continuous learning mindset and interest in cybersecurity technologies.
- Effective communication and mentoring capabilities.
Experience & Qualifications :
- 5 - 7 years of experience in Data Engineering, Platform Engineering, or Cloud Data Solutions.
- Hands-on experience with Azure Data Factory, Databricks, and PySpark.
- Proven experience building and optimizing cloud-based data pipelines.
- Strong understanding of real-time data processing and streaming technologies.
- Experience with Azure cloud services and infrastructure automation tools.
- Knowledge of cybersecurity data platforms, security analytics, or SIEM ecosystems is preferred.
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.
Ways of Working :
- Work closely with Cybersecurity, Infrastructure, and Analytics teams to deliver strategic data engineering solutions.
- Drive continuous enhancement of the enterprise cyber data lake and analytics ecosystem.
- Participate in platform modernization, automation, and operational excellence initiatives.
- Support end-to-end ownership of data onboarding, engineering, monitoring, and optimization activities.
- Contribute to knowledge sharing, technical mentoring, and engineering best practices across the team.
Skills
Data Engineering, Azure, Databricks, PySpark, Azure Data Factory, Kafka, Data Pipeline, ETL, Power BI