Principal Software Engineer- Performance Engineering

Principal Software Engineer, Performance Engineering

Job Summary

We are seeking a Principal Software Engineer, Performance Engineering to embed performance as a first-class engineering discipline across our large-scale applications and databases - shifting performance validation left into design and active development, rather than treating it as a pre-release regression gate. This is a hybrid engineering role: the candidate writes and reviews code, builds performance tooling and instrumentation, and partners directly with feature teams during design and implementation to catch scalability and efficiency issues before they reach production. The ideal candidate is fluent in Linux environments, comfortable authoring both application-level code and test/automation code, skilled in modern load and performance testing frameworks integrated into CI/CD, and capable of deep database performance analysis to identify and resolve bottlenecks at the query, index, and schema level. This role requires a minimum of 10 years of relevant software engineering experience.

Why This Role Is Different

Traditional performance testing runs after code is "done," turning every finding into a costly late-cycle fix. This role exists to move that curve left: performance budgets, load profiles, and profiling hooks become part of design review, code review, and CI - not a separate phase owned by a separate team.

Key Responsibilities

Shift-Left Performance Engineering

• Partner with Product Architects and engineering teams during design and development to define performance/scalability requirements, SLOs, and load profiles before code is written.
• Embed lightweight, code-as-test performance checks (e.g., k6, Gatling, Locust) directly into CI/CD pipelines so regressions surface on every build/PR, not at a downstream test cycle.
• Review production code and architecture for performance anti-patterns (N+1 queries, unbounded loops, inefficient serialization, poor connection/thread pool sizing) as part of the normal code review process.
• Build and maintain reusable performance test harnesses, fixtures, and synthetic data generators that developers can run locally and in CI without specialist hand-holding.

Load, Stress, and Full-Scale Validation

• Design, script, execute, and evaluate performance, load, and stress tests to validate system behavior under expected and peak conditions, using modern load-generation tooling (k6, Gatling, Locust) alongside JMeter/LoadRunner for legacy scenario coverage.
• Develop and maintain scripts to simulate realistic UI and transactional load across applications.
• Use the Linux command line and shell/Python scripting to configure environments, automate test setup/execution/data collection/reporting, and troubleshoot in flight.

Database & Query Performance

• Write and evaluate SQL queries against large-scale databases to support test data preparation and results analysis.
• Perform detailed evaluation of table and index design, query execution plans, and their impact on overall database performance.
• Generate and analyze deep diagnostics (Oracle AWR/ASH, SQL Server Query Store/MDW, or Postgres pg_profile/PoWA) to isolate database-layer bottlenecks.
• Generate and load large volumes of representative test data to support realistic performance validation.

Analysis, Instrumentation & Remediation

• Analyze test and production telemetry (APM, profiling, GC/memory data) to identify root causes and drive optimization - collaborating directly with developers, DBAs, and infrastructure teams on fixes, not just findings.
• Evaluate and tune JVM memory management and garbage collection behavior under load.
• Document performance requirements, findings, and recommendations, and communicate clearly to both technical and non-technical stakeholders.

Required Skills and Experience

• Minimum 10 years of software engineering experience, including a minimum of 6 years focused on performance engineering, performance/load testing, or a closely related discipline.
• Demonstrated ability to write and/or review production application code - this is not a testing-only role.
• Strong proficiency with the Linux command line for navigation, file management, process monitoring, and troubleshooting.
• Semi-advanced scripting ability (Bash/Korn, Python, or Perl) to automate workflows and build reusable tooling.
• Hands-on experience with JMeter and/or LoadRunner for legacy load scenario coverage.
• Hands-on experience integrating performance/load tests into CI/CD pipelines (shift-left practice), using modern frameworks such as k6, Gatling, or Locust.
• Experience with Java memory management, garbage collection, evaluation, and tuning.
• Solid understanding of large-scale database fundamentals: SQL, table/index design, referential integrity, and query performance evaluation.
• Hands-on experience with Oracle AWR/ASH generation and evaluation, or equivalent (SQL Server Query Store/MDW, Postgres pg_profile/PoWA).
• Ability to work with Product Management, Architects, and engineering teams to define performance requirements early in the development lifecycle, and to prioritize competing demands accordingly.
• Familiarity with Kubernetes, Tanzu, or OpenShift for deploying, scaling, and monitoring workloads under test.
• Familiarity with Cassandra or Scylla; performance evaluation and tuning experience in either is a plus.

Preferred (Nice to Have) Skills

• Experience extending JMeter/LoadRunner or custom Java-based test frameworks for runtime transaction generation.
• Experience with APM/observability platforms (Dynatrace, New Relic, Datadog, Grafana) for production-signal-driven performance work.
• Python for data analysis and tooling beyond test automation.
• Basic project management and Agile Scrum experience.

What We Offer

• Ownership of performance as an engineering discipline, not a downstream checkpoint - direct influence on architecture and design decisions.
• Exposure to large-scale, high-throughput fintech systems and modern cloud-native infrastructure.
• Collaborative engineering culture with direct partnership across product, architecture, and infrastructure teams.
• Competitive compensation and benefits package.