rishabh sinha

Google · Santa Clara

Software engineer · ML systems

I build AI
systems that move.

I work on low-latency inference, retrieval, and reliable agentic systems — currently at Google.

01 / Work

Experience

Production ML and distributed systems work across model serving, retrieval, agent orchestration, and scientific computing.

  1. Sunnyvale, CA

    Google

    Software Engineer

    • Delivered Gemini intent-encoder post-training at 100+ QPS and sub-25 ms p99 with QLoRA/SFT on four TPUs.
    • Lowered RAG retrieval p99 by 9% through fan-out DAGs and concurrent embedding-memory reads.
    • Improved LLM serving with prefix-cache reuse across disaggregated prefill and decode, reducing KV recomputation.
    • Hardened serving reliability with safer Borg/Kubernetes rollouts, telemetry, and cell-level rollback.
  2. Seattle, WA

    Amazon Web Services

    Software Development Engineer

    • Architected a Bedrock AgentCore and LangGraph multi-agent orchestrator, cutting inference latency 40% with semantic caching.
    • Built a Kubernetes static-analysis engine around scheduler bin-packing, increasing validation throughput by more than 300%.
    • Improved agentic code-generation success by 25%+ with memory, sandboxed execution, and eval-driven repair loops.
    • Raised agent reliability by 30%+ through deterministic evals, trace replay, and guarded tool execution.
  3. Seattle, WA

    Amazon Web Services

    Software Development Engineer Intern

    • Built a fault-tolerant OAuth-backed integration for synchronizing resource data across 150+ AWS services.
    • Implemented SigV4 authentication and 5 ms LDAP authorization using configuration caching and Guice.
  4. Gaithersburg, MD

    National Institute of Standards and Technology

    Software Engineering Intern

    • Accelerated signal processing by 85% with multiprocessing, NumPy SIMD operations, and parallel execution.
    • Reduced runtime by 45 seconds through thread pooling, profiling, and C++ hot-path optimization.

02 / Upstream

Open source

Merged contributions to the open LLM serving stack.

  1. 01 kvcached Elastic KV-cache sharing across co-located LLMs through GPU virtual memory, integrated with vLLM and SGLang. Contributed fp8 per-token-head KV quantization, allocation rollback on mid-alloc failure, and the cross-layer KV-sharing binding fix. A lifecycle-readiness and failure-propagation workstream is in review with maintainer sign-off. vLLM · SGLang · KV cache
  2. 02 vLLM ecosystem Quantization, routing, and serving components across six repositories in the vLLM org: a per-column quantization-range hoist in compressed-tensors, calibration-failure diagnostics in llm-compressor, a response-format correctness fix in semantic-router, and a controller panic fix in aibrix. Quantization · Routing · Serving
  3. 03 Mooncake KV-cache transfer engine behind disaggregated serving. Migrated the master-service suite to a scenario DSL and fixed an RDMA endpoint leak surfaced by the nightly sanitizer run. C++ · RDMA
  4. 04 TileLang DSL and compiler for high-performance GPU kernels. Contributed eager-frontend correctness fixes and replaced cloudpickle with JSON in the kernel-parameter cache. Compiler · Kernels
  5. 05 llm-d Kubernetes-native distributed inference. Corrected the endpoint-picker configuration schema documentation. Go · Kubernetes

03 / Selected builds

Projects

Measured systems work: throughput, tail latency, scaling, and reproducible evaluation.

  1. 01 SuperRed Composable AI red-teaming across attackers, targets, benchmarks, and explicit threat models. Python · AI Security
  2. 02 Low-Latency LLM Inference 120 tokens/s on Llama-3-8B with 40% lower p99 latency at 32-request concurrency. vLLM · C# · gRPC
  3. 03 RL Hyperparameter Tuner PPO-driven learning-rate tuning for ResNet-18, validated against a 910 img/s JAX ceiling. PPO · PySpark · JAX
  4. 04 SLURM Vision RAG LLaVA-1.5-7B fine-tuning on four A100s with a 1M-vector retrieval index. PyTorch · Slurm · RAG

04 / Papers

Research

Current work on evaluating and understanding the behavior of AI systems.

  1. Preprint

    SuperRed: An AI Red-Teaming and System Security Framework

    Simon Sure · Sarthak Munshi · Rishabh Sinha · Zhun Wang · Dawn Song · ICLR 2027 submission

    A composable framework separating attackers, systems under test, security claims, and controllers. Security domains make adversary trust boundaries explicit, enforced, and sweepable across experiments.

    35
    modules at launch
    4
    swappable components
    Paper Project Source
  2. Under review

    Exposure Is Not Commitment: Double-Stratified Evaluation of Latent Monitors for Tool-Using Agents

    Rishabh Sinha · UC Berkeley · Agents in the Wild @ NeurIPS 2026

    Separates injection exposure from behavioral commitment, in both the evaluation and the training population. Monitors trained on the naive contrast mostly learn exposure; retrained on exposed cases only, the same probes predict the agent’s own unsafe tool call up to two steps early.

    0.85 – 0.91
    commitment AUROC, three models
    2 steps
    lead time, trajectory maximum
    Paper Code & artifacts
  3. Under review

    A Nonzero Final Learning Rate Floor Improves Transfer of Agent-Discovered Schedule Tweaks

    Rishabh Sinha · NeurIPS 2026 main conference

    Tests whether an AutoResearch-discovered schedule change survives controlled decomposition, five-seed reruns, and nearby depth transfer. A 0.05 final learning-rate floor—not warmup—was the robust component in the fixed-budget single-GPU harness.

    4 / 5
    base seeds improved
    5 / 5
    depth-10 seeds improved
    0.000877
    base mean BPB gain
    0.001570
    depth-10 mean BPB gain
    Paper

05 / Education

Education

Computer science, mathematics, data systems, and computational finance.

2026

University of California, Berkeley

Master of Information and Data Science

Part-time professional program GPA 4.0

2025

University of Maryland, College Park

Bachelor’s in Computer Science and Mathematics

Minor in Computational Finance GPA 3.98

06 / Toolkit

Skills

What the work above is built with.

Languages
Python, Go, C++, C, SQL
Inference & serving
vLLM, SGLang, KV-cache systems, prefix caching, disaggregated serving, FP8 and GPTQ quantization
ML & compute
PyTorch, JAX, CUDA, Triton, TPUs
Systems
Kubernetes, Linux, RDMA, gRPC, Ray, Slurm, Docker, OpenTelemetry