Senior GCP DevOps & LLMOps Engineer

Job Description: Senior GCP DevOps & LLMOps Engineer

Role Summary

We are looking for an experienced Senior GCP DevOps & LLMOps Engineer to architect, deploy, and maintain robust infrastructure supporting enterprise-scale Large Language Model (LLM) and Generative AI applications. In this role, you will bridge the gap between traditional DevOps, platform engineering, and modern machine learning operations.

You will lead the automation of Google Cloud Platform (GCP) environments using Terraform , design resilient CI/CD pipelines, and implement production-grade LLMOps architectures —including model serving, inference routing, vector database orchestration, evaluation pipelines, and GPU/TPU resource optimization.

-

Design, build, and maintain production GCP environments using modular, reusable Terraform (and Terragrunt ) code.

-

Manage multi-environment cloud resources across VPC networking, Private Service Connect, Cloud Interconnect, IAM, KMS, and Cloud Storage.

-

Enforce security best practices, least-privilege IAM policies, Workload Identity Federation, and automated compliance auditing.

-

Build automated deployment, scaling, and operational infrastructure for generative AI models hosted on Vertex AI , GKE (Google Kubernetes Engine), or hybrid inference servers.

-

Implement or Understanding end-to-end LLM lifecycle pipelines: prompt management, fine-tuning workloads, automated retrieval-augmented generation (RAG) components, and evaluation harness pipelines.

-

Deploy, manage, and scale vector databases (e.g., Vertex AI Vector Search, Pinecone, Qdrant , pgvector ) with proper indexing, backup, and latency monitoring.

-

Architect and optimize model-serving engines.

-

Build and maintain scalable CI/CD pipelines using Cloud Build , GitHub Actions , or GitLab CI for both microservices and machine learning models.

-

Establish automated testing for infrastructure changes (e.g., tflint , tfsec , terratest ) and model release pipelines (canary deployments, blue/green rollouts, shadow traffic).

-

Package and deploy containerized AI/ML applications using Docker, Artifact Registry, and Helm charts on GKE.

-

Set up deep observability stacks using Google Cloud Monitoring, OpenTelemetry , Prometheus, and Grafana to track standard SRE metrics (SLIs/SLOs, latency, throughput).

-

Implement LLM-specific telemetry: token consumption rates, inference latency, model drift, cost-per-query, and hallucinations/groundedness metrics using tools such as LangSmith , Phoenix/ Arize , or Vertex AI Model Monitoring.

-

Integrate safety guardrails, prompt sanitization, rate limiting, and fallback/resiliency mechanisms across inference endpoints.

-

Monitor and optimize cloud spending for compute-intensive workloads, specifically LLM quota monitoring , spot instances, and dynamic autoscaling.

-

Partner with engineering and product teams to project infrastructure costs and establish budget s and quota guardrails.

-

Experience: 5+ years in DevOps, SRE, or Cloud Platform Engineering with at least 3+ years dedicated to Google Cloud Platform (GCP) .

-

Terraform Expertise: Advanced proficiency with Terraform (state management, reusable module design, workspaces, policy-as-code with Sentinel or OPA).

-

LLMOps / MLOps Background: 1.5+ years of practical experience operationalizing LLMs or ML workloads, including familiarization with Vertex AI (Model Garden, Pipelines, Endpoints) or custom model serving on Kubernetes.

-

Containerization & Orchestration: Strong background in Docker and Google Kubernetes Engine (GKE) , including cluster setup, GPU node pools, ingress controllers, and autoscaling (HPA, KEDA).

-

CI/CD: Hands-on experience configuring automated pipelines with GitOps patterns (e.g., ArgoCD , Flux) or modern CI engines (GitHub Actions, Cloud Build).

-

Scripting & Languages: Solid programming/scripting proficiency in Python and Bash/Shell.

-

Security & Networking: Strong understanding of GCP VPC architecture, private endpoints, service perimeters (VPC-SC), and Secret Manager.

-

Experience deploying agentic frameworks (e.g., LangChain , LlamaIndex , LangGraph , CrewAI ) or Model Context Protocol (MCP) servers in production.

-

GCP Certifications (e.g., Google Cloud Certified Professional Cloud DevOps Engineer, Professional Cloud Architect, or Professional Machine Learning Engineer).

-

Familiarity with distributed model training and high-throughput inference caching strategies (Redis, Semantic Cache).

-

Exposure to regulatory compliance frameworks (SOC2, HIPAA, ISO 27001) in relation to cloud AI systems.

Technical Stack Overview

Domain

Technologies & Tools

Cloud Provider

Google Cloud Platform (GCP)

Infrastructure as Code

Terraform, Terragrunt , OpenTofu , TFLint

Containers & Orchestration

GKE, Docker, Helm, ArgoCD

AI / LLMOps

Vertex AI, vLLM , Triton, Ray, Hugging Face, LangSmith / Phoenix

Vector Search & DBs

Vertex AI Vector Search, Qdrant , Pinecone, Cloud SQL ( pgvector )

CI/CD & Code Quality

GitHub Actions, Cloud Build, GitLab CI, Pre-commit, SonarQube

Observability & Ops

Google Cloud Monitoring, Prometheus, Grafana, OpenTelemetry

Scripting & Automation

Python, Bash