Machine Learning Engineer

Meet the Team The Splunk AI Platform and Services team builds and operates the core runtime and developer infrastructure powering AI across Splunk and Cisco. We run multi-tenant, large-scale LLM inference, VectorDB/RAG, Model Context Protocol (MCP), and AI DevX services across hybrid multi-cloud environments. Sitting at the intersection of Site Reliability Engineering, Distributed Systems, and ML Infrastructure, our mission is to ensure these mission-critical AI workloads remain highly available, scalable, secure, and cost-efficient in production. Your Impact Scale Core AI Services: Operate and optimize multi-tenant LLM inference, model serving, VectorDB/RAG, and MCP platforms across global cloud and GPU clusters. Drive Platform Resiliency: Establish and monitor SLOs, SLIs, automated health checks, and observability dashboards (Prometheus, Grafana, Splunk, OpenTelemetry). Automate Operations & Toil Reduction: Build automation for seamless rollouts, autoscaling, canary deployments, self-healing, and disaster recovery. Optimize GPU & Inference Performance: Tune throughput, latency, model scheduling, batching, caching, and resource allocation using Kubernetes, vLLM, and Ray. Incident Command & Prevention: Lead complex production incident response, conduct 5-Why root-cause analyses, and implement systemic remediations to eliminate repeat outages. Cross-Functional Enablement: Partner with AI researchers, product developers, and platform teams on production readiness reviews (PRRs) and participate in a sustainable on-call rotation. Minimum Qualifications Education & Experience: Bachelor’s degree in CS/Engineering with 5+ years of relevant experience, or Master’s with 3+ years. Software Development: 3+ years of programming experience in Python, Go, or Java building automation, tooling, or production backend services. Cloud-Native Operations: Solid experience running distributed, containerized systems in production using Kubernetes, microservices, and CI/CD. SRE & Observability: Proven background in production operations, distributed troubleshooting, and monitoring stacks (Prometheus, Grafana, Splunk, or OpenTelemetry). Cloud Platforms: Hands-on working knowledge of AWS, Azure, or GCP. Preferred Qualifications LLM & Inference Frameworks: Hands-on experience with vLLM, TensorRT-LLM, Ray Serve, or KServe. GPU Orchestration: Experience managing GPU scheduling, capacity planning, model parallelism, and performance profiling. Stateful & Retrieval Data Stores: Experience operating Vector databases, search engines, or distributed cache layers (e.g., Milvus, Pinecone, Redis). GenAI Architectures: Familiarity with RAG pipelines, model routing, Model Context Protocol (MCP), and agent runtimes. Advanced SRE Design: Track record designing multi-region high-availability systems, automated remediation, and chaos engineering practices. Why Cisco?