Senior Software Engineer, Storage / Infrastructure

The Senior Software Engineer is responsible for researching, evaluating, developing, operating, and optimizing database, storage, and distributed systems that support long-term file, photo, and video storage for Zalo product features.

The role requires strong foundations in software engineering, distributed systems, Linux, and infrastructure, together with the ability to independently investigate and optimize complex technical issues across multiple layers, from applications and database/storage services to operating systems and hardware

What you will do

Technology Research & Proof-of-Concept

• Research and evaluate new technology stacks, databases, storage systems, distributed systems, and infrastructure solutions to address existing technical challenges and future team requirements.
• Build Proof-of-Concept implementations to evaluate technologies before production adoption.
• Evaluate performance, scalability, reliability, compatibility, and operational complexity.
• Design benchmarks and workload tests to validate technical hypotheses and compare alternative solutions.
• Analyze technical trade-offs and recommend solutions suitable for actual workloads and operational conditions.

In-house Storage & Database Systems

• Develop, maintain, and operate in-house database and storage clusters used for long-term file, photo, and video storage for Zalo product features.
• Analyze and resolve issues related to system performance, scalability, reliability, and capacity.
• Implement system changes, upgrades, and optimizations required to support data and workload growth.
• Develop and improve internal tooling and automation for deployment, operation, monitoring, and troubleshooting.
• Investigate complex bottlenecks and issues related to database and storage workloads.

Third-party Distributed Systems

• Operate, maintain, troubleshoot, and tune systems such as ScyllaDB, SeaweedFS, SQL databases, Kafka, and other database, storage, or messaging platforms used by the team.
• Perform cluster deployment, upgrades, configuration changes, scaling, and capacity expansion.
• Monitor production system health, performance, and resource utilization.
• Investigate issues related to latency, throughput, availability, and resource usage.
• Perform root cause analysis for complex issues and propose solutions to prevent recurrence.

Linux, Hardware & Performance Engineering

• Research, operate, troubleshoot, and optimize Linux-based systems, primarily CentOS 7 and Rocky Linux 9.
• Investigate operating system issues affecting production stability, latency, throughput, or resource utilization.
• Analyze performance across multiple layers, from applications and database/storage services to operating systems and hardware, to identify bottlenecks and root causes.
• Tune Linux systems according to workload characteristics, including CPU/scheduling, memory/page cache, network stack, filesystems, disk I/O, resource limits, and kernel parameters.
• Investigate memory pressure, resource contention, I/O latency, network performance, and filesystem behavior.
• Perform benchmarks and workload tests to evaluate the impact of operating system or hardware configuration changes.
• Research and evaluate hardware for database and storage workloads, including RAM, NICs, RAID controllers/configurations, SSDs, NVMe devices, HDDs, and server configurations affecting performance or stability.
• Compare hardware and configuration alternatives based on actual workloads and benchmark results.
• Recommend configurations by considering performance, reliability, capacity, and cost trade-offs.

Monitoring, Reliability & Troubleshooting

• Build and improve monitoring, metrics, logging, and alerting for assigned systems.
• Monitor performance, resource utilization, capacity, and overall system health.
• Troubleshoot complex production issues involving applications, databases, operating systems, networks, or hardware.
• Perform root cause analysis and recommend corrective actions for significant incidents.
• Improve tooling, automation, and configuration to increase system reliability and reduce operational workload.

Cross-team Coordination

• Coordinate with product teams, infrastructure teams, and other engineering teams to resolve dependencies, support requirements, and technical issues during implementation.
• Proactively track and follow up on infrastructure and cross-team dependencies to support product delivery timelines.
• Collaborate with relevant teams to investigate and resolve issues during integration, deployment, and production operation.
• Keep track of infrastructure and related system changes that may affect the systems under the team's responsibility.

What you need

• Strong foundation in software engineering and backend/distributed systems.
• Good understanding of replication, partitioning, consistency, availability, fault tolerance, and failure recovery.
• Experience working with database, storage, or data infrastructure systems.
• Strong Linux system administration, troubleshooting, and performance tuning skills.
• Experience with CentOS, RHEL, Rocky Linux, or equivalent Linux distributions.
• Experience operating high-availability production systems with large workloads.
• Experience with technologies such as ScyllaDB/Cassandra, Kafka, SQL databases, SeaweedFS, or equivalent distributed storage systems.
• Understanding of how CPU, RAM, NICs, RAID controllers, SSDs, NVMe devices, and HDDs affect database and storage workloads.
• Experience with monitoring, metrics, logging, and observability.
• Programming or scripting skills for automation, internal tooling, and Proof-of-Concept development.
• Strong troubleshooting, analytical, and problem-solving skills.