No history yet

Low-Latency Token Optimization

The Agentic Latency Challenge

Agentic loops introduce a stateful orchestration layer, but they also create a new performance bottleneck: cumulative latency. Each turn in a ReAct loop is a full inference pass, compounding wait times. The 'Iron Triangle' of Cost, Quality, and Latency becomes acute here. While multi-step reasoning boosts quality, it drives up both token count and user-perceived delay.

We need to optimize the entire loop, not just a single call. Two key metrics define the user experience in this context: (TTFT) and (ITL). TTFT measures the time from query to the start of the response, signaling that the agent is working. ITL measures the speed of the streaming output itself. In an agentic workflow, a long TTFT can occur while the model plans or calls a tool, even if the final text generation is fast.

The goal is to minimize TTFT for the initial response and maintain a low, consistent ITL for each step in the agent's execution trace.

Routing and Parallelization

A powerful strategy is 'Small Model First' routing. Not every task in a loop requires a frontier model. A smaller, faster model (like a 7B or 8B parameter model) can handle simpler intermediate steps, such as classifying user intent, routing to a tool, or formatting parameters. This reduces both cost and latency for the preparatory stages of the loop.

The frontier model (e.g., GPT-4, Claude 3 Opus) is then invoked only for the core reasoning or generation task where its capabilities are essential. This creates a cascade where the bulk of the work is offloaded to cheaper, faster models.

Furthermore, if a planning step identifies multiple independent tool calls, they can be executed in parallel. For example, if an agent needs to fetch a user's calendar and the current weather, these two API calls do not depend on each other. Dispatching them concurrently significantly reduces the total time spent waiting for external data.

import asyncio

async def get_weather(location):
    # API call to weather service
    print("Fetching weather...")
    await asyncio.sleep(1) # Simulate network delay
    return {"weather": "Sunny"}

async def get_calendar(user_id):
    # API call to calendar service
    print("Fetching calendar...")
    await asyncio.sleep(1.5) # Simulate network delay
    return {"events": ["Team Meeting at 3 PM"]}

async def parallel_tool_calls():
    # Schedule both calls to run concurrently
    weather_task = asyncio.create_task(get_weather("SF"))
    calendar_task = asyncio.create_task(get_calendar(123))

    # Wait for both to complete
    weather_result = await weather_task
    calendar_result = await calendar_task
    
    return {**weather_result, **calendar_result}

# Running this would execute both API calls at the same time,
# finishing in ~1.5 seconds instead of 2.5 seconds.
# results = await parallel_tool_calls()

Managing Context Overhead

As an agent loop progresses, the context window fills with prior thoughts, actions, and observations. This ever-growing history increases token costs and can eventually exceed the model's context limit. Effective context management is non-negotiable.

One direct approach is setting a token budget. After each loop, you can prune the history, removing the oldest turns to stay within the limit. A more nuanced method is context pruning, where you selectively remove less relevant turns or tool outputs based on an importance score, perhaps determined by a smaller, faster LLM.

Context Compression: Summarizing earlier turns to reduce token load

Summarization is a powerful form of context compression. Instead of retaining the full, verbose history, a separate LLM call can periodically create a concise summary of the progress so far. This summary replaces the raw history, freeing up thousands of tokens while preserving the essential state needed for the agent to continue its task. This is analogous to how human memory consolidates episodic details into semantic knowledge.

Ready to test your knowledge on optimizing agentic workflows?

Quiz Questions 1/5

What is the primary performance bottleneck specifically created by the stateful, multi-turn nature of agentic loops?

Quiz Questions 2/5

The 'Small Model First' routing strategy involves using a smaller, faster model for simpler tasks like classifying intent or formatting parameters, reserving the larger frontier model for the core reasoning task.

By combining intelligent routing, parallelization, and aggressive context management, you can build agents that are not only powerful but also fast and cost-effective, resolving the inherent tensions of the Iron Triangle.