•Building and evolving the LLM API platform, from the moment a request hits the edge to the response that goes back out
•The Python→Go rewrite: building services in Go alongside the existing Python stack, without compromising reliability
•Streaming infrastructure: SSE for Chat Completions and Responses, long-lived connections, clean handling of cancellations and client disconnects
•Routing and failover: weighted provider chains, retries with backoff, and latency-driven admission control across model deployments
•Protocol compatibility: keeping the API faithful to the OpenAI spec across inference servers (vLLM, SGLang, Dynamo) that each behave slightly differently
•The commercial layer: metering, billing, and rate-limiting, engineered to the same bar as the model APIs themselves
•Performance and reliability for a multi-tenant platform serving long-context agentic and coding workloads
•Observability (logging, metrics, tracing), so the team can debug production confidently and quickly
•Integration and load testing: the harness that lets us ship fast without breaking customer-facing APIs
•Close partnership with the Inference and MLOps teams on production readiness for new model launches
•Design-first, documentation-first engineering culture within the team: RFCs before code, decisions written down, no tribal knowledge
WHAT WE'RE LOOKING FOR
•3+ years building production backend systems
•Strong Python in production: asyncio, FastAPI or similar, and services you've built and operated under real load
•Working knowledge of Go, or a strong willingness to learn it: you'll work in both languages through the rewrite
•Experience building low-latency, multi-tenant backend services
•Hands-on experience with streaming systems: SSE or WebSockets, long-lived connections, backpressure
•Strong PostgreSQL and Redis fundamentals: schema design, query performance, caching, and rate limiting
•Comfort running services on Kubernetes in production
•An ownership mindset: you test before you ship, debug production with metrics, logs, and traces, and write the RCA afterwards
