LLM Production Infrastructure — Research Brief (2026-10-03)

Key Developments

• Shared KV caches can silently corrupt LLM answers in production

  • What changed: Academic audit found vLLM and SGLang shared KV-cache connectors cause cross-tenant accuracy collisions.
  • Why it matters: Multi-tenant clusters pooling caches for cost savings may be quietly trading accuracy for throughput.
  • Sources: [1]

• Shared cache timing lets attackers reconstruct other tenants' prompts

  • What changed: Academic audit found shared KV-cache connectors in vLLM and SGLang leak cross-tenant prompts via timing.
  • Why it matters: Multi-tenant model-hosting clusters pooling caches for cost savings may be quietly leaking tenant prompt content.
  • Sources: [1]

• OpenAI lets enterprises spend committed budget on a rival's hosting platform

  • What changed: OpenAI's new marketplace lets enterprise customers route traffic to Baseten-hosted open models using existing OpenAI spend commitments.
  • Why it matters: Procurement teams gain contractual flexibility to mix open and closed models without separate vendor agreements.
  • Sources: [3], [4]

• Researchers propose bidding-based pricing for scarce inference capacity

  • What changed: A UC Berkeley-led team designed an inference auction letting users bid for priority service, validated against SGLang without sacrificing latency.
  • Why it matters: Signals a path beyond flat-rate SLA tiers toward demand-responsive pricing enterprises should track in future vendor contracts.
  • Sources: [5]

Notable Papers / Models / Tools

Technical Deep-Dive

A significant infrastructure finding this cycle is a structural flaw in how production LLM serving stacks compose caching layers. Modern deployments typically pair an inference engine's local prefix cache — which already tracks adapter, precision, and sharing-domain identity — with a separate shared KV-cache tier (LMCache- or Mooncake-style systems) that exists specifically to let a fleet of replicas reuse computed attention state across requests. The Griffith/NUS/Macquarie team's audit found that this shared tier commonly keys entries only by token content and coarse model metadata, which erases provenance and lets identical tokens under incompatible computational or sharing contexts collide — a gap the authors term provenance-blind reuse [1].

The empirical scope is what elevates this from a theoretical nitpick to an operational risk. A source audit of three vLLM connectors confirms the structural omission, while runtime experiments reproduce it across vLLM and two SGLang releases, 12 models from 7 families (0.5B–32B), and over 160 configurations [1]. The consequences split into two failure classes. The first is silent correctness decay: cross-adapter collisions reduce accuracy from 0.94 to 0.64, and incompatible KV representations reduce reasoning accuracy to zero [1] — meaning a serving stack can return a confidently wrong answer with no error signal, because the cache "hit" looks identical to a correct one. The second is a genuine security leak: salt omission enables 93% prompt identification from timing [1], letting a co-tenant on a shared cache infer another tenant's prompt content purely by observing response latency. For a bank running a shared multi-tenant inference cluster — whether self-hosted or via a MaaS vendor — this is a direct hit against both model-risk (unflagged accuracy regressions) and data-confidentiality controls.

The paper's proposed fix, a formalized KV provenance contract requiring shared keys to be injective over a declared set of computational and sharing dimensions, is notable for being practical rather than purely theoretical: the authors report an implementation in vLLM and SGLang across three cache paths that eliminates unsafe reuse while preserving legitimate sharing, with hit-path latency changes remaining within sub-millisecond bounds and below run-to-run variation — the safety fix appears close to free in the cases studied. The limitation worth flagging for procurement teams: this is a single-paper audit of specific connector versions, and the fix requires each shared-cache operator to adopt the provenance-contract discipline — there is no indication yet that LMCache, Mooncake, or hyperscaler-managed shared-cache offerings have incorporated it. A companion independent study on replicated 27B inference [8] reinforces the broader pattern that cache "hit" signals in shared-serving architectures are not reliable correctness proxies — this looks like the opening of a new research and audit front on serving-stack caching, not an isolated incident.

Landscape Trends

• [LLM Production Infrastructure × Safety, Assurance & Governance] The KV-cache provenance findings [1] extend a thread first flagged in the 2026-09-09 brief (CPU-cache detokenization leaks, contention-degraded side-channel detection): cache-layer side channels are shifting from a performance footnote to a recurring audit requirement, and the new accuracy-collapse finding adds a model-risk dimension beyond pure confidentiality.
• [LLM Production Infrastructure × Enterprise GenAI Adoption] OpenAI's marketplace move [3],[4] is a direct infrastructure-layer response to the multi-model fragmentation flagged in the 2026-09-30 Enterprise Adoption brief (81% of Global 2000 firms running three-plus model families); portable committed spend reduces one FinOps pain point but extends a bank's OpenAI contract indirectly into Baseten's infrastructure, creating new third-party vendor-risk surface.
• Inference pricing is moving from fixed tiers toward dynamic, demand-responsive allocation. The Berkeley auction mechanism [5] and Fireworks' cache-aware FireRouter, which claims over 50% cost reduction by routing routine turns to open models [11], both point toward per-request cost optimization replacing flat-rate tiers — worth tracking for budget-predictability implications even though neither is yet a standard commercial offering.
• Self-hosting economics research is consolidating around vendor-backed, production-integrated tooling rather than academic benchmarks alone. QATFactory's direct export path into vLLM and llama.cpp [2] continues a pattern from the 2026-09-15 and 2026-09-27 briefs (py-kvcache, AsymFlow, KVSET) of serving-stack research increasingly shipping as usable connectors, lowering integration cost for new efficiency techniques.
• Regulated-deployment network controls are becoming native platform features rather than custom engineering. Microsoft Foundry's public-preview egress controls for hosted agents [12] — enforcing allow/deny rules on outbound agent traffic via policy rather than code — show managed-hosting platforms absorbing VPC-isolation patterns that financial-services teams previously had to build themselves.

Vendor Landscape

Modal Labs is reportedly closing a $750 million funding round led by Accel at a $15.75 billion valuation, more than tripling the AI infrastructure startup's valuation from four months earlier [9]; coverage notes rivals Baseten, Fireworks, and Fal also in talks for sharply higher valuations, suggesting a broader funding surge across managed-inference vendors rather than a Modal-specific story. Modal paired this with product news at its inaugural Runtime conference, launching generally-available RDMA-connected multi-node GPU clusters plus VM Sandboxes, Sandbox Sidecars, and Sticky Sessions for agent workloads [10] — a Tier 2, vendor-sourced capability change worth watching for independent validation before treating it as a mature self-hosting alternative to hyperscaler capacity commitments. Separately, Fireworks AI launched FireRouter with Opus, a cache-aware router that shifts routine requests to open models while reserving Claude Opus for harder turns, claiming over 50% session-cost reduction at comparable accuracy [11] — a vendor claim not yet independently verified.

Sources

1. arXiv:2609.38706, "Preserving Provenance in Shared KV Caches for LLM Serving" (Sep 30, 2026) — https://arxiv.org/abs/2609.38706 [Tier 1 — academic, Griffith University / National University of Singapore / Macquarie University]
2. arXiv:2609.39223, "QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs" (Sep 30, 2026) — https://arxiv.org/abs/2609.39223 [Tier 1 — industry research, Together AI]
3. Baseten, "Announcing our partnership with OpenAI" (Sep 29, 2026) — https://www.baseten.co/blog/baseten-openai-partnership/ [Tier 2 — vendor]
4. The Register, "OpenAI apes AWS Marketplace while Anthropic remains stubbornly Microsoftian" (Sep 30, 2026) — https://www.theregister.com/columnists/2026/09/30/openai-apes-aws-marketplace-while-anthropic-remains-stubbornly-microsoftian/5300056 [Tier 1 — independent journalism]
5. arXiv:2609.40070, "Inference Auctions" (Sep 30, 2026) — https://arxiv.org/abs/2609.40070 [Tier 1 — academic, UC Berkeley / TTIC / Google Research]
6. arXiv:2610.01218, "AGO AI Quality Gate: Evidence-First Release Decisions for Retrieval-Augmented Generation" (Oct 1, 2026) — https://arxiv.org/abs/2610.01218 [Tier 2 — industry-affiliated, Protom Group S.p.A.]
7. Zenodo, "Fixed-K Speculative Decoding vs. EAGLE-3: A Like-for-Like, Energy-Measured Comparison in vLLM and SGLang" (2026) — https://openalex.org/W7214401052 [Unaffiliated preprint, unverified]
8. arXiv:2609.15021, "Shared KV Caching for Replicated 27B Inference: Correctness Failures and Performance Boundaries" (Sep 2026) — https://arxiv.org/pdf/2609.15021 [Affiliation unclear, moderate confidence]
9. TechCrunch, "Source: Inference provider Modal Labs closing in on $750M round at $15.75B valuation" (Sep 28, 2026) — https://techcrunch.com/2026/09/28/source-inference-provider-modal-labs-closing-in-on-750m-round-at-15-75b-valuation/ [Tier 1 — independent journalism]
10. Modal, "Runtime Roundup: VM Sandboxes, Multi-node clusters, and more" (Oct 1, 2026) — https://modal.com/blog/runtime-product-update-sandbox-endpoints [Tier 2 — vendor]
11. Fireworks AI, "Introducing FireRouter with Opus" (Sep 28, 2026) — https://fireworks.ai/blog/introducing-firerouter-with-opus [Tier 2 — vendor]
12. Microsoft Foundry Blog, "Control where your hosted agent connects with network egress in Foundry Agent Service" (~Sep 18–19, 2026) — https://devblogs.microsoft.com/foundry/egress-controls-hosted-agent/ [Tier 2 — vendor]