rishabh sinha

Google · Santa Clara

Software engineer · ML systems

I build AI
systems that move.

I work on low-latency inference, retrieval, and reliable agentic systems — currently at Google.

01 / Work

Experience

Production ML and distributed systems work across model serving, retrieval, agent orchestration, and scientific computing.

  1. Sunnyvale, CA

    Google

    Software Engineer

    • Delivered Gemini intent-encoder post-training at 100+ QPS and sub-25 ms p99 with QLoRA/SFT on four TPUs.
    • Lowered RAG retrieval p99 by 9% through fan-out DAGs and concurrent embedding-memory reads.
    • Improved LLM serving with prefix-cache reuse across disaggregated prefill and decode, reducing KV recomputation.
    • Hardened serving reliability with safer Borg/Kubernetes rollouts, telemetry, and cell-level rollback.
  2. Seattle, WA

    Amazon Web Services

    Software Development Engineer

    • Architected a Bedrock AgentCore and LangGraph multi-agent orchestrator, cutting inference latency 40% with semantic caching.
    • Built a Kubernetes static-analysis engine around scheduler bin-packing, increasing validation throughput by more than 300%.
    • Improved agentic code-generation success by 25%+ with memory, sandboxed execution, and eval-driven repair loops.
    • Raised agent reliability by 30%+ through deterministic evals, trace replay, and guarded tool execution.
  3. Seattle, WA

    Amazon Web Services

    Software Development Engineer Intern

    • Built a fault-tolerant OAuth-backed integration for synchronizing resource data across 150+ AWS services.
    • Implemented SigV4 authentication and 5 ms LDAP authorization using configuration caching and Guice.
  4. Gaithersburg, MD

    National Institute of Standards and Technology

    Software Engineering Intern

    • Accelerated signal processing by 85% with multiprocessing, NumPy SIMD operations, and parallel execution.
    • Reduced runtime by 45 seconds through thread pooling, profiling, and C++ hot-path optimization.

02 / Upstream

Open source

Merged contributions to the open LLM serving stack.

  1. 01 kvcached Elastic KV-cache sharing across co-located LLMs through GPU virtual memory, integrated with vLLM and SGLang. Contributed fp8 per-token-head KV quantization, allocation rollback on mid-alloc failure, and the cross-layer KV-sharing binding fix. A lifecycle-readiness and failure-propagation workstream is in review with maintainer sign-off. vLLM · SGLang · KV cache
  2. 02 vLLM ecosystem Quantization, routing, and serving components across six repositories in the vLLM org: a per-column quantization-range hoist in compressed-tensors, calibration-failure diagnostics in llm-compressor, a response-format correctness fix in semantic-router, and a controller panic fix in aibrix. Quantization · Routing · Serving
  3. 03 Mooncake KV-cache transfer engine behind disaggregated serving. Migrated the master-service suite to a scenario DSL and fixed an RDMA endpoint leak surfaced by the nightly sanitizer run. C++ · RDMA
  4. 04 TileLang DSL and compiler for high-performance GPU kernels. Contributed eager-frontend correctness fixes and replaced cloudpickle with JSON in the kernel-parameter cache. Compiler · Kernels
  5. 05 llm-d Kubernetes-native distributed inference. Corrected the endpoint-picker configuration schema documentation. Go · Kubernetes

03 / Selected builds

Projects

Measured systems work: throughput, tail latency, scaling, and reproducible evaluation.

  1. 01 SuperRed Composable AI red-teaming across attackers, targets, benchmarks, and explicit threat models. Python · AI Security
  2. 02 Low-Latency LLM Inference 120 tokens/s on Llama-3-8B with 40% lower p99 latency at 32-request concurrency. vLLM · C# · gRPC
  3. 03 RL Hyperparameter Tuner PPO-driven learning-rate tuning for ResNet-18, validated against a 910 img/s JAX ceiling. PPO · PySpark · JAX
  4. 04 SLURM Vision RAG LLaVA-1.5-7B fine-tuning on four A100s with a 1M-vector retrieval index. PyTorch · Slurm · RAG

04 / Papers

Research

Current work on evaluating and understanding the behavior of AI systems.

  1. Preprint

    SuperRed: An AI Red-Teaming and System Security Framework

    Simon Sure · Sarthak Munshi · Rishabh Sinha · Zhun Wang · Dawn Song · ICLR 2027 submission

    A composable framework separating attackers, systems under test, security claims, and controllers. Security domains make adversary trust boundaries explicit, enforced, and sweepable across experiments.

    35
    modules at launch
    4
    swappable components
    Paper Project Source
  2. Accepted

    Exposure Is Not Commitment: Double-Stratified Evaluation of Latent Monitors for Tool-Using Agents

    Rishabh Sinha · UC Berkeley · Agents in the Wild @ NeurIPS 2026

    Separates injection exposure from behavioral commitment, in both the evaluation and the training population. Monitors trained on the naive contrast mostly learn exposure; retrained on exposed cases only, the same probes predict the agent’s own unsafe tool call up to two steps early.

    0.85 – 0.91
    commitment AUROC, three models
    2 steps
    lead time, trajectory maximum
    Paper Code & artifacts
  3. Under review

    A Nonzero Final Learning Rate Floor Improves Transfer of Agent-Discovered Schedule Tweaks

    Rishabh Sinha · NeurIPS 2026 main conference

    Tests whether an AutoResearch-discovered schedule change survives controlled decomposition, five-seed reruns, and nearby depth transfer. A 0.05 final learning-rate floor—not warmup—was the robust component in the fixed-budget single-GPU harness.

    4 / 5
    base seeds improved
    5 / 5
    depth-10 seeds improved
    0.000877
    base mean BPB gain
    0.001570
    depth-10 mean BPB gain
    Paper

05 / Notes

Writing

Mini projects from the exploration phase of Neel Nanda’s MATS 12 stream, October 2026.

  1. 01 Banning board words from a Codenames spymaster’s reasoning gpt-oss-120b respelled the banned words instead of dropping them, and the ban cost about two thirds of the gain from reasoning. CoT · Monitoring
  2. 02 Inkling’s reply opener Forcing “I can assure you” after traces that plan to admit being an AI raised clean human claims from 9 to 132 of 300 replies. Model forensics
  3. 03 Thought Branches with a matched control Hint transplant curves jump at single sentences, and the late hint effect may come from how the released traces were selected. CoT faithfulness
  4. 04 Gaming an evaluation without saying so No route produced runs to test silent gaming, but blocking talk of a detector undid much of a disguised copy. Eval awareness
  5. 05 Do AI judges recognise their own work? A “your own” label raised ratings of identical transcripts, while the true author made no detectable difference. Monitoring

06 / Education

Education

Computer science, mathematics, data systems, and computational finance.

2026

University of California, Berkeley

Master of Information and Data Science

Part-time professional program GPA 4.0

2025

University of Maryland, College Park

Bachelor’s in Computer Science and Mathematics

Minor in Computational Finance GPA 3.98

07 / Toolkit

Skills

What the work above is built with.

Languages
Python, Go, C++, C, SQL
Inference & serving
vLLM, SGLang, KV-cache systems, prefix caching, disaggregated serving, FP8 and GPTQ quantization
ML & compute
PyTorch, JAX, CUDA, Triton, TPUs
Systems
Kubernetes, Linux, RDMA, gRPC, Ray, Slurm, Docker, OpenTelemetry