[RFC] LMCache & Agentic Application/Benchmark/Workflow Traces

Goal

Collect a set of agentic applications and workflows to demonstrate that LMCache is suitable for running agentic applications/workflows in terms of context reuse, ultimately reducing TTFT (Time to First Token) and ITL (Inter-Token Latency).

Motivation

We want to showcase that LMCache is not only limited to CPU offloading. The plan is to collect real traces to prove that LMCache can speed up inference in agentic settings. Agentic workloads typically involve multiple iterative LLM calls within a session; this pattern benefits from cache reuse. This issue is a long-lasting tracker and a call for help.

Plan

A new folder will be added:

lmcache/examples/agents/

Each subfolder will correspond to a benchmark, application, or workflow:

examples/agents/<benchmark_or_application_name>/
 ├── README.md          # Brief intro to the benchmark/application/workflow and how to run it
 ├── trace.jsonl        # Example converted trace
 └── converter.py       # (Optional) script to convert raw logs/outputs to the unified trace format

Unified Trace Format

Different repos log LLM calls, function calls, intermediate results, and faults in different ways. We only care about the LLM calls that reach the vLLM server, as those are relevant to LMCache.

trace.jsonl

{"timestamp": 0, "input": "input prompt in raw string", "output": "output response in raw string", "session_id": 0}
{"timestamp": 123, "input": "input prompt in raw string", "output": "output response in raw string", "session_id": 0}

• timestamp: relative timestamp from 0.
• input / output: raw strings for each LLM call.
• session_id: if one query to an agent executes N rounds and incurs N LLM calls, all entries share the same session_id. This enables overlapping different users/sessions to make the dataset more condensed. session_id is experimental and optional; the key fields are timestamp, input, and output.

Under each benchmark/application/workflow folder, include an example converted trace and, optionally, a converter.py to help others convert the repo’s raw outputs to the unified format.

Analysis and Visualization

After obtaining unified traces, I will provide an analysis script to study the relationship between RAM pool size and hit rate for both:

1. Prefix string match (LMCache case).
2. Longest common substring match (CacheBlend case).

The physical meaning of the graphs is: given a unified RAM pool (GPU/CPU/Disk) for KV-cache storage, how much hit rate can be achieved.

For example:

• If 20 GB is the spare GPU RAM after loading the model, that is the maximum hit rate achievable if only vLLM prefix caching happens in GPU.
• If 200 GB is the maximum CPU RAM, then the hit rate difference between 200 GB and 20 GB is the LMCache gain: gain = f(200) − f(20).

We can also annotate the x-axis with: no prefix caching → GPU caching → CPU caching → disk caching → unlimited space. This helps guide best-practice configuration for LMCache.

Example figures (not good ones though): Prefix Cache Hit Rate vs Pool Size (LMCcache)

Substring Cache Hit Rate vs Pool Size (CacheBlend)

Purpose of Calculating Hit Rate

If we have the hit rate, we can estimate performance gain.

Let:

• Tp = time to prefill,
• To = time for a cache hit and offload,
• assume Tp = 10 × To.
• HRl: hit rate for lmcache; HRv: hit rate for vllm

Then:

ttft_lmcache = Tp(1 - HRl) + To*HRl = 10To - 10To*HRl + To*HRl = 10To - 9To*HRl
ttft_vllm    = Tp(1 - HRv) + To*HRv = 10To - 10To*HRv + To*HRv = 10To - 9To*HRv

ttft_lmcache / ttft_vllm
= (10To - 9To*HRl) / (10To - 9To*HRv)
= (10To - 9To*(HRv + HRdiff)) / (10To - 9To*HRv)
= 1 - 9To*HRdiff / (10To - 9To*HRv)

Alternatively, you can set up a machine and replay the traces directly to observe TTFT and ITL differences.

Call for Contribution

Help is needed in this thread. The request is to run popular agentic benchmarks/applications/workflows, each within a folder under lmcache/examples/agents. Each folder should contain a simple README about the benchmark/application/workflow, instructions for running it, and the converted trace. Optionally include converter.py to help others convert raw outputs to the unified format.

We are currently running:

• Open Deep Research: https://github.com/langchain-ai/open_deep_research
• Terminus 1 Agent: https://www.tbench.ai/terminus

Any additional traces from agentic benchmarks/applications/workflows would be greatly appreciated. Ideas and comments are welcome.

Coordination

@yamazakihirofumi has relevant past experience and will work on this thread as discussed in the community meeting.

This issue will serve as a long-term tracker for collecting, converting, and analyzing agentic traces to evaluate LMCache’s effectiveness beyond CPU offloading and to quantify its impact on TTFT and ITL.