News

AWS authors detail how they cut agent tokens up to 71% on large JSON

Published 4 min readAI-assisted · human approved
TL;DR

An AWS Machine Learning Blog post describes strategies AWS authors used for a Flightscape agent on Amazon Bedrock AgentCore handling 20 MB+ JSON. They report up to 71% fewer tokens and up to 80% lower average latency.

Why it matters

If your agent pushes large structured payloads through tool calls, this shows a concrete pattern: keep bulky data in S3, store only references in session memory, and hand the model summaries. The figures are the authors' own benchmark results on one 23 MB payload, so measure on your data.

Image from AWS Machine Learning Blog
Image: AWS Machine Learning Blog. It belongs to its owner; see our image policy.

Original source

AWS published a post on 17 November 2026 describing a Flightscape agent built on Amazon Bedrock AgentCore that handles 20 MB+ JSON payloads. The authors report token use cut by up to 71% and average latency by up to 80% across their benchmark runs.

What happened

In a post on the AWS Machine Learning Blog, authors Pallav Gupta and Pascal Acquaah explain how they built a Training Schedule Optimization agent for Flightscape, an airline operations platform for planning crews, flights and operations. The agent verifies crew training schedule assignments against business rules and answers queries about assignments, scheduling and resource allocation.

The first version delivered the needed functionality but was slow and expensive. According to the post, it had 40+ tools and megabyte-sized JSON inputs, and most calls averaged more than 15 seconds. Each tool invocation pushed its entire payload into the model's context, which inflated token counts and cost. With bigger inputs, the context window was exhausted before the agent could finish reasoning.

The authors say the combined optimizations cut token consumption by up to 71% and average latency by up to 80% across benchmark runs on a representative 23 MB payload. These are the authors' own results from one workload.

The architecture

The agent is built with LangGraph, an open source framework from LangChain, and runs on Amazon Bedrock AgentCore. It uses an Amazon Bedrock model for reasoning and AgentCore memory to persist chat across turns. It works over three JSON documents: the optimizer's request, its response, and a validation report from a business rules validator. The post notes that Amazon Bedrock Guardrails can be applied to validate and filter the agent's responses.

The agent's state also carries basic statistics and cache data to support richer queries.

Three composable strategies

The post presents three composable strategies and says Strategy 1 runs at the start and end of each turn, while Strategies 2 and 3 run each time a tool returns a result. The text describes Strategy 1 in detail; for the other two it gives only this turn-level overview.

Strategy 1: persist lightweight session state, offload heavy artifacts

The authors found that large payloads were stored too close to conversational state. They moved heavy artifacts (request JSON, response JSON, validation data and other large derived payloads) to Amazon S3. AgentCore memory now keeps only stable references to those objects. Tools load the heavy data on demand through the references instead of on every turn.

Two functions bracket each turn:

  • hydrate_state() restores lightweight session state from AgentCore memory at the start of a turn, rebuilding the live graph state by deserializing messages, re-parsing raw payloads into domain objects and restoring every reference key.
  • dehydrate_state() writes heavy payloads to the S3ArtifactStore at the end of a turn and saves only references to AgentCore memory.

What happens when a tool returns a result

Per the turn overview, when a tool returns an oversized list or dictionary, the agent stores the full result in Amazon S3 and passes the model a compact summary plus a handle. The model can then request more detail with get_result_page_tool().

Key details

  • Published 17 November 2026 on the AWS Machine Learning Blog, tagged Advanced (300).
  • Reported results: up to 71% fewer tokens and up to 80% lower average latency on a representative 23 MB payload, per the authors' benchmark runs.
  • Prerequisites listed: an AWS account with Amazon Bedrock access and a supported model, an S3 bucket for artifact storage, AgentCore memory enabled, a LangGraph-based agent (or equivalent) with tool calling, and Python 3.12+ with boto3, langchain-core and langgraph.
  • Companion code is available in a GitHub repository linked from the post.

Try it

The core of Strategy 1 is a persisted-state schema that keeps messages inline and heavy data as S3 keys. This is the schema the post shows being written to AgentCore memory:

class PersistedState(TypedDict, total=False):
    """Serialized session state for AgentCore Memory.
    Heavy fields (request, responses) are stored in S3 and referenced by key.
    Lightweight fields are stored inline.
    """
    messages: List[Any]  # LangChain-serialized messages
    request_ref: Optional[str]  # S3 key for raw request JSON
    responses_ref: Optional[str]  # S3 key for raw responses JSON list
    cache_data_ref: Optional[str]  # S3 key for cache_data payload
    validation_report_ref: Optional[str]  # S3 key for validation_report payload
    basic_stats_ref: Optional[str]  # S3 key for basic_stats payload

The live GraphState keeps the parsed domain objects tools need and mirrors these reference fields, so later turns can locate their artifacts.

Why it matters

The pattern applies to any agent whose tools return large structured data. Keeping that data out of the context window and session memory lowers tokens per turn, and summaries with retrieval handles let the model fetch detail only when needed. For related cost thinking, see OpenBeat's coverage of measuring Amazon Bedrock agent cost per task and the next-generation AgentCore runtime.

The 71% and 80% figures are maxima from the authors' benchmarks on a single payload size. Results on your workload will depend on payload shape, tool count and how often the model needs the full data.

Checked claims

ClaimStatusEvidence
The authors report up to 71% lower token consumption and up to 80% lower average latency on a representative 23 MB payload in their benchmark runs.confirmedthese optimizations reduced token consumption by up to 71% and cut average latency by up to 80% across our benchmark runs on a representative 23 MB payload
The agent is built with LangGraph and runs on Amazon Bedrock AgentCore.confirmedThe agent is built with LangGraph , an open source framework by LangChain, and runs on Amazon Bedrock AgentCore
When a tool returns an oversized list or dictionary, the full result goes to S3 and the model receives a compact summary with a handle, with more detail available via get_result_page_tool().confirmedthe agent stores the full result in Amazon Simple Storage Service (Amazon S3) and passes the model a compact summary with a handle.
Strategy 1 runs at the start and end of each turn; Strategies 2 and 3 run each time a tool returns a result.confirmedStrategy 1 runs at the start and end of each turn, and Strategies 2 and 3 run each time a tool returns a result.
Heavy artifacts moved to Amazon S3, with AgentCore memory persisting only references to them.confirmedAgentCore memory then persists only stable references to these objects, keeping the session state lightweight.

Found a mistake? Use Report above; an editor reads every one. Editorial policy

Question for readersHow do you keep large tool outputs from filling your agent's context window, and what trade-offs have you hit with summaries or retrieval handles?

Discussion · 0

No comments yet. Start the discussion.

Sign in to join the discussion.