memory consumption and model routing between AI's especially with cache tokens

Key points on memory consumption, KV/token cache, and model routing:

1. What consumes memory in LLM serving

   • Two dominant consumers: model weights (fixed) and KV cache (grows with tokens, batch, concurrency). [[1, 2]]
   • Example: LLaMA 7B in FP16 is ~14 GB of weights; at 4K tokens, KV cache can be ~2 GB for batch size 1, and grows linearly with sequence length and batch size. [[3, 4]]
   • At scale, KV cache can rival or exceed weight memory, becoming the main bottleneck for long contexts and many concurrent users. [[5, 2]]

2. KV (token) cache basics

   • Transformer stores key/value vectors per layer per token so it doesn’t recompute attention for past tokens, turning attention cost from quadratic to roughly linear in sequence length. [[6, 7]]
   • Memory for KV cache ≈ num_layers × num_heads × head_dim × bytes_per_element × tokens × batch_size (constant per token for a given model → overall linear in tokens and batch). [[8]]
   • Prefill phase: compute K/V for the whole prompt and fill cache; Decode phase: reuse KV cache to generate new tokens; decode is strongly memory‑bandwidth bound due to KV reads/writes. [[5, 2]]

3. Why cache tokens dominate memory

   • Long‑context agent sessions (tens of thousands of tokens) cause KV cache to grow fastest of all components in GPU memory. [[5, 9]]
   • Fixed-slot KV allocation (pre‑allocating for max context) wastes memory badly; e.g., allocating for 4K max context but using only 100 tokens wastes ~97.5% of reserved cache space. [[10]]

4. Optimizations to reduce KV memory

   • Paged / virtualized KV (e.g., vLLM PagedAttention): divide KV cache into pages; allocate on demand; reuse and evict pages, similar to OS virtual memory. This avoids large contiguous allocations and reduces fragmentation and waste, enabling longer contexts and more users on the same GPU. [[10, 11]]
   • Offloading KV: move KV to CPU RAM or NVMe SSD when GPU memory is tight; trades latency/bandwidth for larger effective context or higher concurrency. [[12, 5, 13]]
   • Quantizing/compressing KV: use lower precision or sparsity-aware layouts for KV to cut bytes per element, directly reducing per-token KV footprint. [[7, 8]]
   • Architectural tricks (e.g., sliding window, GQA): limit effective attention window per layer or reduce number/size of heads, shrinking KV per token. [[14, 11]]
   • Eviction policies: drop least-important tokens (e.g., lowest attention scores) from cache, especially on edge or constrained devices. [[15, 8]]

5. Model routing with KV‑aware caches

   • In multi‑model or multi‑pod setups, routing is critical to reuse KV and avoid recomputation.
   • KV‑cache‑aware routing (e.g., Red Hat’s llm-d over vLLM):
     • Maintains a global index of KV blocks across serving pods. [[16]]
     • For a new request, finds the pod with the longest matching prefix (shared tokens) already cached in GPU memory. [[16]]
     • Achieves high cache‑hit rates (e.g., ~87% for prefix-heavy workloads), faster time-to-first-token, and less redundant compute. [[10, 16]]
   • Application-level prefix caching (shared system prompts, shared RAG preambles) plus KV-aware routing massively cuts both compute and KV growth across users. [[1, 2]]

6. Routing between different models (“multiple AIs”)

   • When orchestrating several models (e.g., base LLM, code LLM, RAG LLM, embeddings):
     • Each model typically has its own KV space; KV is not reusable across different architectures or tokenizers.
     • Memory planning must consider sum of (weights + peak KV) across all concurrently active models. Running multiple local models may require partial CPU offload or strict KV limits per model to fit within GPU VRAM. [[4]]
   • Practical strategies:
     • Pin heavy base model on GPU; route short, specialized tasks to smaller models that have smaller KV footprints.
     • Use keep‑alive TTLs and KV-slot caps per model to prevent one long session from monopolizing KV capacity when several models share a GPU. [[4]]
     • Combine token-aware load balancing (route long contexts to GPUs with more KV headroom) with KV-aware routing within each model (route to the pod that already holds the matching context). [[3, 1]]

7. Impact on throughput and concurrency

   • With naive KV handling, bigger contexts and more users quickly exhaust GPU memory, forcing tiny batch sizes and low throughput. [[1, 17]]
   • Systems that pair efficient KV management + intelligent routing (e.g., vLLM with paged KV plus KV-aware routing and continuous batching) have shown multi‑x throughput gains over naive implementations while using the same GPUs. [[2, 11]]

Summary:

• KV (token) cache, not just model weights, is the principal limiter for memory usage, concurrency, and context length in multi-model (“multi‑AI”) systems. [[5, 1]]
• Memory- and cache-aware routing (per-model, KV-prefix-aware, and GPU headroom-aware) plus KV paging, offloading, compression, and eviction are the core techniques to keep memory consumption under control while routing between models efficiently. [[10, 16, 8, 2]]

The Architecture of Memory: Stateful Routing, KV Cache Optimization, and Cross-Model Communication in LLM Inference

The operational paradigm of large language models (LLMs) has fundamentally shifted from isolated, stateless queries to complex, stateful, and iterative interactions. As enterprise applications increasingly adopt multi-agent workflows, long-context conversational interfaces, and retrieval-augmented generation (RAG), the infrastructure supporting these computational models must grapple with unprecedented memory demands. In contemporary LLM serving, raw compute availability—measured in floating-point operations per second (FLOPs)—is rarely the primary bottleneck. Rather, the strict limits of system throughput are dictated by memory bandwidth and the sophisticated management of intermediate inference states, most notably the Key-Value (KV) cache [cite: 1, 2, 3].

The transition toward compound AI systems, where multiple specialized models collaborate dynamically on shared contextual data, has necessitated a complete reimagining of the underlying inference infrastructure. To sustain low latency, maximize concurrent batch sizes, and prevent astronomical compute costs, the industry is moving aggressively beyond localized single-node memory management. The modern AI serving stack relies on disaggregated prefill and decode architectures, distributed hierarchical KV cache pools, state-aware request routing, and emerging techniques for direct semantic communication between disparate model architectures operating within a shared latent space.

The Physics of LLM Serving and the KV Cache Bottleneck

To understand the constraints of modern AI serving, one must examine the autoregressive nature of transformer-based LLMs. Text generation is fundamentally repetitive and computationally asymmetrical. The process is divided into two distinct execution phases. Initially, the model processes a complete input sequence during the prefill phase, computing the necessary contextual representations in a highly parallelized, compute-bound operation. Following this, the model enters the decode phase, generating output one token at a time [cite: 1, 3, 4]. During decoding, each newly generated token is appended to the context window and fed back into the neural network for the subsequent step, creating a sequential, memory-bound workload [cite: 4, 5].

The attention mechanism, which empowers the model to capture deep contextual relationships across the sequence, relies heavily on the computation of query ($Q$), key ($K$), and value ($V$) projections. Before a model can process a sentence, the input is broken into discrete tokens, each mapped to high-dimensional embedding vectors. To inject context, these embeddings are multiplied by learned weight matrices to produce the $Q$, $K$, and $V$ representations. The scaled dot-product attention is then calculated mathematically as:

$$Attention(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_{model}}}\right) V$$

Without systemic optimization, calculating the attention scores for a newly generated token $t$ would require the model to recompute the full attention matrix for all $t-1$ preceding tokens. Because this computation must occur at every layer of the transformer, the naive approach yields an $O(n^2)$ computational complexity relative to sequence length, which is entirely impractical for production deployment [cite: 2, 6, 7].

The universal solution to this computational bottleneck is KV caching. Instead of recalculating historical representations, the system stores the $K$ and $V$ matrices of all previously processed tokens in the GPU's High Bandwidth Memory (HBM). By deliberately trading memory capacity for computational speed, KV caching reduces the complexity of generating subsequent tokens from quadratic to linear. The model only needs to compute the $Q$, $K$, and $V$ vectors for the single newest token, and computes its attention scores against the cached historical vectors [cite: 7, 8].

However, this indispensable efficiency mechanism introduces a severe memory scaling problem. The size of the KV cache grows multi-linearly with sequence length, transformer layer depth, batch size, and the model's hidden dimension [cite: 2, 8]. For example, a 70-billion parameter model operating with a batch size of 32 and an 8,192-token context window demands approximately 40 to 50 GB of VRAM solely for the KV cache [cite: 2]. This requirement frequently exceeds the memory footprint of the model weights themselves, creating a strict upper bound on system concurrency [cite: 2, 9, 10].

Memory Fragmentation and Allocation Paradigms

Historically, LLM serving engines allocated contiguous blocks of GPU memory for the maximum possible sequence length of an incoming request. Because actual sequence lengths are highly variable and unpredictable, this naive allocation strategy resulted in catastrophic internal and external memory fragmentation. Serving infrastructure routinely wasted between 60% and 80% of available GPU memory, leaving compute units idle while waiting for memory to clear [cite: 2, 3].

The PagedAttention Revolution

The first wave of systemic optimization successfully addressed this memory fragmentation. Borrowing core concepts from operating system virtual memory management, frameworks like vLLM introduced PagedAttention. By partitioning KV caches into fixed-size, non-contiguous memory blocks, PagedAttention decouples the logical sequence of tokens from their physical storage in GPU memory [cite: 2, 11, 12].

A centralized memory manager maps continuous logical blocks to non-continuous physical pages, allocating new pages dynamically only when needed during the decoding phase. This architectural shift reduced memory waste to under 4%, effectively doubling to quadrupling the number of concurrent requests a server could handle without hardware upgrades [cite: 2, 9]. PagedAttention rapidly became the industry standard for maximizing the throughput of isolated, independent queries [cite: 11, 13, 14].

RadixAttention and Automatic Prefix Caching

While PagedAttention brilliantly solved fragmentation for isolated requests, it did not inherently optimize for shared contexts across multiple users. In dense production environments—such as customer support chatbots utilizing extensive system instructions, agentic swarms leveraging identical tool schemas, or RAG pipelines prepending the same retrieved documents—recomputing the KV cache for these shared prefixes is highly inefficient [cite: 15, 16, 17].

This systemic redundancy catalyzed the development of automatic prefix caching, most notably implemented as RadixAttention within the SGLang framework [cite: 11, 13, 15]. RadixAttention restructures the KV cache into a radix tree, enabling zero-cost memory sharing for identical token prefixes across concurrent requests. Unlike flat caching architectures, a radix tree gracefully manages conversation forks, allowing multiple agents to explore alternative branching paths without recalculating the foundational attention weights of the shared prompt [cite: 11, 14, 15].

The architectural decision between these engines largely depends on the workload's prefix overlap ratio. If over 60% of requests share a common prefix, SGLang's RadixAttention yields a 70% to 90% cache hit rate, drastically reducing Time-to-First-Token (TTFT) by entirely skipping the compute-bound prefill phase for the shared segments [cite: 15, 18]. Furthermore, SGLang achieves up to 2.5x faster structured output generation (e.g., producing strict JSON) through a compressed finite state machine approach that overlaps mask generation [cite: 14, 15]. Conversely, for workloads dominated by highly unique, non-overlapping prompts, vLLM's continuous batching and broader ecosystem support remain highly competitive, operating within 5% of SGLang's raw unconstrained throughput [cite: 13, 15, 18].

Internal Compression: Cross-Layer Sharing and Outlier Tracing

Beyond sophisticated memory allocation, deep learning research has focused heavily on reducing the raw dimensional footprint of the KV cache itself through cross-layer sharing and dynamic compression, addressing the reality that not all cached tokens possess equal semantic utility.

Standard transformer architectures maintain entirely independent KV caches for each attention layer. However, recent analyses utilizing Centered Kernel Alignment (CKA) reveal that the dominant singular vectors of KV caches are highly aligned across adjacent layers [cite: 19]. This exposes a massive, unexploited redundancy within the model's intermediate states.

Methods like xKV leverage this alignment by introducing post-training cross-layer factorization (CLF) to extract a shared basis of singular vectors. This mathematical process allows multiple adjacent transformer layers to share a compact token basis, achieving up to 8x compression in KV cache footprint with negligible accuracy loss (under 3%) on rigorous long-context benchmarks such as RAG and LongBench [cite: 19, 20]. By aggressively compressing the cache in this manner, inference systems can increase batch sizes within the same strict VRAM budget, reducing attention latency by up to 3.6x and achieving end-to-end throughput speedups exceeding 4x [cite: 19].

In a similarly counterintuitive breakthrough, the KVSharer framework observed that sharing dissimilar KV caches between specific layers preserves model performance better than sharing highly similar ones [cite: 21]. KVSharer operates as a plug-and-play mechanism without requiring architectural retraining. It identifies optimal sharing strategies by analyzing Euclidean distances between layer representations on a calibration dataset, sorting them in descending order. It then allows the direct replacement of KV caches during the forward pass, yielding high layer-wise compression while maintaining structural output integrity [cite: 21, 22].

Addressing Tokenizer Misalignment and Quantization Failures

Alongside architectural sharing, data-level compression via quantization remains critical for production viability. Quantizing the KV cache from standard FP16 precision down to INT8 or INT4 halves the memory footprint but traditionally introduces unacceptable accuracy degradation [cite: 7, 8]. A persistent challenge in KV quantization is the presence of "outlier tokens"—highly specific tokens exhibiting atypically small Key vector magnitudes that disproportionately skew the numerical boundaries during the quantization process [cite: 23].

The Outlier Tokens Tracing (OTT) methodology provides a tuning-free mechanism to resolve this. OTT dynamically detects and isolates these sensitive outlier tokens during the autoregressive decoding phase, preserving their full-precision representations while allowing the vast majority of the cache to be heavily quantized. This hybrid numerical approach enables up to 6.4x memory savings while accelerating inference by 2.3x, presenting a highly practical solution for deploying LLMs in resource-constrained hardware environments [cite: 23].

The Disaggregation Imperative and Hierarchical Storage

As context windows push past the 128,000-token threshold, the structural limitations of executing both prefill and decode phases on the same GPU node have become glaringly apparent [cite: 8, 24, 25]. The two phases possess diametrically opposed computational profiles. Prefill processes thousands of tokens simultaneously; it relies heavily on dense matrix multiplication and is strictly compute-bound, bottlenecked only by GPU FLOPs [cite: 3, 5, 26]. In contrast, decode processes just one token per sequence. It requires fetching massive KV cache tensors from memory to compute a single token's attention, making it strictly memory bandwidth-bound [cite: 5, 27].

When batched together on a single homogeneous GPU cluster, prefill requests inevitably monopolize compute resources. This causes severe latency spikes and unpredictable jitter for concurrent decoding requests [cite: 27, 28].

The industry's structural response is Prefill-Decode (PD) Disaggregation: physically separating prefill and decode workloads onto dedicated hardware clusters [cite: 29, 30, 31]. Compute-dense GPUs handle the initial prefill, while high-memory-bandwidth GPUs manage the subsequent decoding loops.

The Network Bottleneck and Remote Direct Memory Access (RDMA)

Disaggregation fundamentally relies on the ability to migrate the generated KV cache from the prefill node to the decode node [cite: 31]. If the network transfer latency across the cluster exceeds the time saved by isolation, disaggregation fails, erasing the prefill-side gain [cite: 30]. Consequently, the network fabric and intermediate storage layers have become just as vital to LLM serving as the GPUs themselves.

This architectural requirement has birthed advanced distributed KV cache engines like Mooncake, InfiniStore, and LMCache, which treat the network as an extension of the local memory bus.

Mooncake, utilized by Moonshot AI for the Kimi service, implements a KVCache-centric disaggregated architecture. It leverages underutilized CPU, DRAM, and SSD resources across GPU clusters to build a unified cache pool. By implementing prediction-based early rejection and balancing effective throughput with service-level objectives, Mooncake achieves up to a 525% throughput increase in long-context workloads by eliminating redundant prefills [cite: 29, 32, 33].

InfiniStore, an open-source high-performance KV store developed by ByteDance, acts as a dedicated RDMA-based KV cache server. By utilizing GPU Direct RDMA (GDR) and zero-copy data paths over InfiniBand or RoCE (RDMA over Converged Ethernet), memory tensors are moved directly from GPU High Bandwidth Memory (HBM) to the Network Interface Card (NIC). This entirely bypasses the host CPU and system RAM, utilizing complex scatter-gather lists to efficiently move non-contiguous memory blocks [cite: 34, 35, 36].

LMCache operates as a powerful caching middleware connector that augments engines like vLLM. It provides a persistent, hierarchical storage system spanning GPU VRAM, CPU pinned DRAM, local NVMe, and remote storage layers (like Redis or InfiniStore). When a vLLM instance processes a prompt, the LMCache connector hashes the token sequences [cite: 37, 38, 39]. If a match is found in remote storage, the KV blocks are fetched directly into GPU memory via high-bandwidth channels like NVIDIA's NIXL transfer engine [cite: 24, 25, 38]. Communication between vLLM and LMCache occurs over a ZeroMQ (ZMQ) dealer/router pattern, allowing seamless integration [cite: 40].

The performance implications of these tiered storage solutions are staggering. Testing on massive 130,000-token context windows with a 32 GB KV cache demonstrated that integrating LMCache with vLLM dropped TTFT from over 11 seconds to a mere 1.5 seconds, delivering a 10x improvement in prefill performance and a 15x overall throughput improvement for multi-round question answering [cite: 25, 37, 39]. Architectures like ROCm AMD Infinity Context (AIC) similarly integrate network-attached storage using NFS over RDMA, dramatically lowering infrastructure costs by storing cold KV cache on cheaper NVMe tiers instead of exorbitant GPU HBM [cite: 24].

Cost, Complexity, and Capability-Based Routing

While intra-node memory optimizations drive performance, application-level model routing drives financial viability. AI workloads within consumer and enterprise applications feature a vast spectrum of task complexity. The computational requirements for simple intent classification, content moderation, or short conversational acknowledgments are fundamentally different from those required for nuanced, multi-step logical reasoning [cite: 41, 42].

Directing every subtask to a massive frontier model (e.g., a 100B+ parameter model) is financially reckless and architecturally wasteful. AI model routing is the practice of analyzing incoming requests and dynamically directing them to the most appropriate language model based on complexity, latency requirements, and cost tolerances [cite: 41, 43, 44].

Model routers sitting between the application and provider endpoints evaluate tasks to enforce capability-based routing. For example, a 7-billion parameter model might cost $0.10 per million tokens and respond almost instantaneously, whereas a highly capable frontier model might cost $30 per million tokens with significantly higher latency. Because 70% to 80% of subtasks in a typical conversational AI turn can be handled by small, open-weight models, aggregate cost per turn drops by 80% to 95% in a routed architecture [cite: 41, 43, 44].

In controlled evaluations, robust routing platforms like RouteLLM (using matrix factorization) achieved 85% cost savings while maintaining 95% of the baseline frontier model performance [cite: 44, 45]. Academic proposals like SCORE maximize response quality while rigorously respecting user-specified cost and latency constraints, continuously adapting routing decisions as real-time system loads shift [cite: 46]. For applications scaling beyond 100,000 daily active users, the cost differential between a routed and non-routed architecture can routinely exceed hundreds of thousands of dollars per month, making the engineering overhead of a routing layer highly justifiable [cite: 41].

Stateful Routing: Balancing Cache Affinity with Cluster Load

At the infrastructure tier, routing decisions extend far beyond determining which model to use; they must determine exactly which physical GPU node should process the request. In traditional stateless web services, round-robin load balancing is an elegant and optimal algorithm. For LLM serving, however, round-robin is highly destructive [cite: 16, 47].

Because KV caches are inherently stateful and localized to specific GPU instances, distributing requests blindly causes catastrophic cache misses. If a user submits a complex multi-turn prompt containing a large RAG document to Node A (which pays the heavy prefill compute cost to build the cache), and their subsequent query is routed to Node B, Node B possesses no memory of the prior context and must recompute the entire prefix from scratch [cite: 16, 47]. At scale, this "load laundering" phenomenon wastes tens of thousands of GPU hours daily on entirely redundant prefill computations [cite: 16, 48].

Prefix-Aware and KV-Aware Routing Protocols

To counteract this, the industry has aggressively adopted Prefix-Aware Routing, where load balancers function as distributed cache managers. Routers such as the GKE Inference Gateway (utilizing the Endpoint Picker), RayServe's PrefixCacheAffinityRouter, and the llm-d Router actively monitor the KV cache state across the entire fleet [cite: 47, 49, 50].

When a request arrives, the router hashes the prompt prefix and queries its index to locate which worker node currently possesses the longest consecutive cached sequence [cite: 48, 50]. The request is intentionally routed to that specific replica. For workflows heavily dependent on shared contexts—such as standardized system prompts or RAG retrieval blocks—prefix-aware routing flips cache hit rates from random chance (e.g., 25% across 4 nodes) to highly deterministic (approaching 90%), resulting in 57x faster response times on warm hits and doubling aggregate throughput on identical hardware [cite: 48, 51, 52].

Resolving the Trade-off: Cache Affinity vs. Load Balancing

However, strict cache-affinity routing introduces a severe new vulnerability: prompt hotspotting. In real-world workloads, prompt popularity is highly skewed [cite: 53]. If a specific system template or popular tool schema becomes universally requested, a strict cache-aware router will stubbornly send all traffic to the single node holding that cache. This overloads the node's decode capacity, generating massive queue delays, while perfectly capable neighboring nodes sit idle [cite: 53]. A backend with a perfect cache hit but an overloaded queue will deliver worse end-to-end latency than a cold node with ample compute headroom [cite: 16, 54].

Advanced orchestration layers must reconcile this precise tension between cache locality and load distribution. Frameworks like AIBrix (by ByteDance) utilize load-aware routing that monitors both KV cache utilization and GPU hardware states, relying on a distributed KV cache runtime to transfer memory tensors between nodes dynamically to balance loads [cite: 55, 56, 57].

Recent academic designs like DualMap offer highly elegant algorithmic solutions. DualMap breaks the traditional trade-off by operating outside a single mapping space. It maps each incoming request to two candidate instances using independent hash functions—leveraging the "power of two choices" principle [cite: 53, 58]. The scheduler then intelligently selects the superior candidate based on real-time system states. If an instance holding the ideal cache is overloaded and risks violating the Time-to-First-Token (TTFT) Service Level Objective (SLO), DualMap dynamically falls back to load-aware scheduling, migrating the request to the less-loaded second candidate, thereby mitigating hotspots [cite: 53, 59]. This dual-hash-ring approach improves effective cluster capacity by up to 2.25x compared to singular routing strategies, ensuring robust performance under highly skewed traffic [cite: 58, 60].

Cross-Model KV Reutilization in Compound AI Systems

While prefix caching excels at sharing contexts within instances of the same model, compound AI ecosystems frequently deploy distinct models for highly specialized tasks. Historically, if two different models needed to process the same extensive RAG document or collaborative context, both had to individually compute and store their own distinct KV caches, representing massive computational redundancy [cite: 61, 62].

Recent research has dismantled the assumption that KV caches are strictly bound to a specific set of weights. DroidSpeak is the first distributed inference system engineered to enable KV cache reuse across different LLMs, provided they share the same foundational architecture (e.g., a base Llama-3 model and a task-specific fine-tuned Llama-3-Instruct model) [cite: 61, 63].

Naively sharing a KV cache between models with different weights results in catastrophic generation failure, akin to decoding a video with an incompatible codec [cite: 63, 64]. However, DroidSpeak utilizes offline profiling to reveal that only specific "critical layers" are highly sensitive to weight variations [cite: 64, 65]. By selectively recomputing only these critical layers while seamlessly reusing the KV caches for the remaining layers across the network, DroidSpeak enables multi-model setups to accelerate prefill by 3.1x and increase throughput by 4x with negligible quality degradation [cite: 61, 64, 65].

Similarly, PrefillShare takes a radical architectural approach to disaggregated serving. It factorizes language models into a shared, frozen prefill module and separate, task-specific, fine-tuned decode modules [cite: 28]. Multiple specialized decoding models can consume the exact same KV cache generated by the base prefill module. When integrated with a routing and cache-handoff mechanism, this completely eliminates inter-model computational redundancy, delivering 4.5x lower latency in multi-model agent workloads [cite: 28, 66].

Collective Caching for Multi-Agent Synchronized Rounds

In massive multi-agent simulations—such as generative agent societies—execution occurs in synchronized rounds. A central scheduler gathers outputs from all agents and broadcasts the combined conversational context back to them. This "All-Gather" communication pattern creates staggering KV cache redundancy, as every agent's prompt contains the exact same shared historical blocks appended to a unique private sequence [cite: 67, 68].

Standard prefix caching fails catastrophically here because the private, agent-specific history diverges before the shared global context block, breaking the consecutive prefix rule required by engines like vLLM [cite: 67]. TokenDance resolves this inefficiency by operating on the multi-agent round collectively rather than optimizing requests individually [cite: 67, 69]. TokenDance groups the requests, shares the rotary position embeddings (RoPE) computations across the swarm, and introduces a Master-Mirror Diff-Aware storage system. It encodes sibling caches as block-sparse diffs against a single master copy, compressing KV storage by 94% and supporting nearly 3x more concurrent agents on local hardware while meeting strict latency SLOs [cite: 67, 68, 69].

Semantic Communication: The Cache-to-Cache Paradigm

The most profound evolution in multi-model interaction challenges the very medium of communication. In traditional agentic systems, models communicate by generating plain text tokens. This presents three severe bottlenecks: text generation is painfully slow (requiring token-by-token decoding), it creates a strict information bottleneck that discards deep latent semantics, and natural language is inherently ambiguous (e.g., losing structured formatting intent) [cite: 70, 71].

The Cache-to-Cache (C2C) paradigm bypasses text entirely, enabling direct semantic communication between entirely different LLM architectures through their internal memory [cite: 70, 72]. C2C utilizes a specialized neural network called a Cache Fuser [cite: 70, 71].

Instead of generating a text instruction, a Source Model projects its internal KV cache directly into the representation space of the Target Model [cite: 70]. The Cache Fuser employs a projection module to align the distinct mathematical formats of the two models, a dynamic weighting system to filter information, and an adaptive gating mechanism to selectively inject the semantic vectors into the appropriate target layers [cite: 70, 71, 73].

Research utilizing "latent bridges via vector translations" demonstrates that transferring semantics directly between models (e.g., Llama-2-7B sharing meaning with Mistral-7B) avoids explicit text generation entirely [cite: 73, 74]. This paradigm yields notable improvements, achieving up to 14.2% higher accuracy on collaborative tasks and running at twice the speed of text-based exchange [cite: 73, 74]. Interestingly, bidirectional evaluation reveals a 2.01:1 transfer asymmetry, indicating that general-purpose base models yield representations that are far more transferable than their instruction-tuned variants [cite: 74]. This signals a profound paradigm shift: future AI clusters may communicate purely in high-dimensional latent vectors, sharing profound, unambiguous mathematical meaning rather than serialized text.

Security Vulnerabilities and Privacy-Aware Caching

The widespread adoption of Automatic Prefix Caching (APC) in multi-tenant cloud environments has inadvertently introduced a severe security vulnerability. Because retrieving a cached prefix is orders of magnitude faster than recomputing it, APC creates a highly measurable timing side-channel [cite: 75, 76].

In sophisticated probing attacks such as PromptPeek, an adversary can probe the shared infrastructure by sending carefully crafted token sequences [cite: 10, 77]. By measuring the Time-to-First-Token (TTFT), the attacker can deterministically infer whether their guessed prefix was already present in the shared KV cache. If the TTFT is anomalously low (a cache hit), it confirms that another tenant recently processed that exact token sequence. Through iterative guessing and timing analysis, attackers can execute word-by-word prompt stealing, recovering proprietary system instructions, private financial data, or sensitive personally identifiable information (PII) from other concurrent users [cite: 76, 78].

Traditional defenses require strict user-level cache isolation, effectively disabling global APC and destroying the performance benefits of shared routing (degrading TTFT by up to 40% and plummeting throughput) [cite: 78, 79, 80]. To preserve system efficiency while ensuring strict security, systems researchers have developed hybrid, privacy-aware KV cache managers.

CacheSolidarity tackles this by monitoring global cache reuse patterns, proactively flagging suspicious, highly iterative sharing behaviors typical of probing attacks. It selectively isolates prefixes, restricting their reuse only when active threat heuristics are triggered, enabling up to 70% higher cache reuse compared to sledgehammer isolation defenses [cite: 80, 81].

Similarly, SafeKV integrates privacy classification directly into the inference runtime. It utilizes a three-tier asynchronous detection pipeline to evaluate the semantic sensitivity of incoming prompts without blocking generation. SafeKV employs a unified radix-tree index that dynamically segregates public, safe KV entries—which are globally shared to maximize throughput—from sensitive KV entries, which are mathematically confined to private, isolated memory pools [cite: 78, 79]. This selective isolation successfully mitigates 94% to 97% of timing-based side-channel attacks while reducing the cache-induced TTFT overhead from a debilitating 50% down to roughly 11% [cite: 78, 79, 80].

By intertwining secure hardware routing with sophisticated cache segregation, modern LLM infrastructure can maintain the exponential efficiency gains of shared memory while preserving the strict data privacy required for enterprise adoption.

Sources:

1. dev.to
2. introl.com
3. runpod.io
4. towardsai.net
5. perplexity.ai
6. medium.com
7. towardsai.net
8. arxiv.org
9. pomogomo.com
10. ndss-symposium.org
11. rajatpandit.com
12. zansara.dev
13. plainenglish.io
14. techsy.io
15. gigagpu.com
16. acethecloud.com
17. medium.com
18. spheron.network
19. arxiv.org
20. emergentmind.com
21. medium.com
22. researchgate.net
23. arize.com
24. amd.com
25. vastdata.com
26. openreview.net
27. arxiv.org
28. arxiv.org
29. github.io
30. arxiv.org
31. researchgate.net
32. usenix.org
33. arxiv.org
34. lmcache.ai
35. github.com
36. medium.com
37. arxiv.org
38. lmcache.ai
39. lmcache.ai
40. lmcache.ai
41. inworld.ai
42. medium.com
43. mindstudio.ai
44. digitalapplied.com
45. augmentcode.com
46. harvard.edu
47. spheron.network
48. digitalocean.com
49. ray.io
50. llm-d.ai
51. llm-d.ai
52. redhat.com
53. arxiv.org
54. modular.com
55. vllm.ai
56. readthedocs.io
57. github.io
58. arxiv.org
59. arxiv.org
60. openreview.net
61. microsoft.com
62. arxiv.org
63. arxiv.org
64. usenix.org
65. themoonlight.io
66. dou.ac
67. arxiv.org
68. researchgate.net
69. arxiv.org
70. the-decoder.com
71. marktechpost.com
72. arxiv.org
73. arxiv.org
74. arxiv.org
75. arxiv.org
76. promptfoo.dev
77. ndss-symposium.org
78. arxiv.org
79. arxiv.org
80. researchgate.net
81. arxiv.org