OpenSearch Lead Developer

We are looking for a Search & Platform Engineer to join us on a 6month contract, taking ownership of our production clinical search infrastructure. This is a hands-on backend engineering role at the heart of a RAG-based system that serves medical information to healthcare professionals. You will own the OpenSearch infrastructure, the intelligent query pipeline, and the data ingestion systems that keep the platform current. The platform is in development, the codebase is well-structured, and the documentation is thorough. You will be set up to move quickly from day one. This role will sit alongside a dedicated AI Evaluation & RAG Engineer. You own the infrastructure and retrieval layer; they own quality assessment and answer generation tuning. Close collaboration between the two roles will be essential.

The candidate will have responsibilities across the following functions:

Search infrastructure and retrieval:

• Own and operate the OpenSearch infrastructure: index design, mappings, custom analysers, and query tuning across BM25 lexical, kNN vector, and hybrid.
• RRF search methods.
• Maintain and improve the intelligent query pipeline: spell correction, abbreviation expansion, query routing (lexical/vector/hybrid), and multi-intent query decomposition.
• Manage multilingual indices (monograph-{lang}-v1 vector indices using text-embedding-3-large) and maintain consistency across languages.
• Own index versioning, re-indexing workflows, and rollback procedures.

Data ingestion and pipelines:

• Own the ingestion pipeline end-to-end: XML content from the LEGO CMS through to OpenSearch lexical and vector indices.
• Maintain ingestion reliability, index versioning, and re-indexing workflows.
• Ensure data quality, synonym management, and SNOMED CT ontology currency.

Service reliability and observability:

• Maintain the FastAPI service (Python 3.11+), including rate limiting, API key auth, structured JSON logging, and Prometheus metrics.
• Monitor and respond to pipeline degradation events across each stage (rewrite, route, embed, decompose, generate), each of which degrades independently.
• Support Kubernetes deployments and GitHub Actions CI/CD pipelines across dev.

Requirements:

• Strong, production-grade Python (3.10+)? FastAPI or similar, async patterns, pytest, CI-gated test suites.
• Deep hands-on experience with OpenSearch or Elasticsearch: Index design, custom analysers, BM25 tuning, query DSL, mappings, aggregations.
• Practical experience with vector search and hybrid retrieval: kNN/ANN indices, embedding pipelines, RRF or similar fusion approaches.
• Data pipeline engineering: Ingestion, transformation, index management, structured and semi-structured content at scale.
• Comfortable owning a production system with real users: graceful degradation, observability, on-call realities.
• Familiarity with SNOMED CT or clinical ontologies is a plus, not a requirement.

Desirable:

• AWS experience - IAM roles, ECS/EKS, CloudWatch, Lambda, API Gateway.
• Docker and Kubernetes - comfortable reading and editing deployment manifests.
• Multilingual search - per-language indices, language-aware analysers.
• Frontend or REST API integration experience (the platform serves a React/Next.js frontend via a unified search endpoint).

The tech stack:

• Search and Vector: OpenSearch, BM25 kNN (text-embedding-3-large), RRF hybrid, SNOMED CT.
• Backend: Python 3.11 FastAPI, pydantic-settings, uv.
• Infrastructure: Docker, Kubernetes, AWS (ECS/EKS, IAM, Lambda, API Gateway), GitHub Actions.
• Observability: Prometheus, structured JSON logging, correlation IDs, 8 alert rules.
• Ingestion: XML from LEGO CMS? OpenSearch lexical + vector index pipeline.
• Frontend API: REST? Unified POST /search + auxiliary GET endpoints (mobile, autocomplete, SNOMED).

Skills

ANN, Elasticsearch, FastAPI, OpenSearch, Python, kNN, vector