It is 6:40 on a Monday morning and the supply-chain ops agent is halfway through its weekly run. Built with Google’s Agent Development Kit and backed by Gemini, it runs on Cloud Run. It has already pulled 212 open orders from the order system, flagged 31 that cannot ship from their assigned warehouse, and called the inventory service to move stock for 18 of them. It is drafting the fourth of nine emails to suppliers about short shipments when Cloud Run recycles the instance.
The run is gone. When the scheduler retries, the agent starts from the first prompt. Gemini reasons its way to a slightly different plan, the inventory tool moves stock that was already moved, and three suppliers get the same email twice. By 9:00, the ops lead is reconciling warehouse counts by hand and fielding confused replies from vendors.
Nothing here is a bug in the agent’s logic. It is a mismatch between how agents run and how containers live.
Why container restarts break agent runs
An ADK agent is a loop. The model reads the conversation, decides to call a tool, gets the result, and decides again. A real operational task can take dozens of iterations and many minutes, with each tool call touching an external system.
The platforms underneath were designed for short, stateless requests. On Cloud Run, instances scale to zero when traffic drops, requests have a maximum timeout, and the platform can shut down an instance whenever it decides to rebalance. On GKE, nodes get drained during cluster upgrades, autoscalers remove underused nodes, and spot or preemptible nodes can be reclaimed with very little notice.
All of these events are normal from the platform’s point of view. The problem is that the agent’s progress lives in process memory. The loop counter, the list of tool results so far, and the knowledge of which side effects already happened all disappear with the container.
Why the common workarounds fall short
Most teams reach for one of three fixes first.
Persist the session. ADK has a session service, and you can back it with a database so conversation history survives a restart. But a saved transcript is not a resumable execution. Nothing in the session tells a fresh process “you were inside iteration 14, the reallocation tool returned success, and the email tool was called but its result never came back.” Something still has to drive the loop forward from the right point, and that something is gone.
Retry the whole request. A Cloud Tasks queue or a Pub/Sub push subscription will happily redeliver a failed request. But the retry starts the agent from scratch. Because the model is non-deterministic, the second run may take a different path, and every tool call that already succeeded runs again. For tools that move inventory or send email, that causes real damage.
Make the instance live longer. Raising timeouts, setting minimum instances, or pinning agents to on-demand nodes makes the problem rarer. It does not remove it. Deployments, node upgrades, and out-of-memory kills still occur, and you pay for idle capacity to make failures rarer rather than survivable.
Each treats the symptom. The agent loop still has no durable record of what it has done.
The general approach: durable execution for the agent loop
Treat an agent run like a workflow rather than a request. Durable workflow engines have handled this for years in order processing and payments, and the same principles apply.
- Record every step. Each LLM decision and each tool call is persisted to a durable store as it completes, as an append-only history.
- Resume by replay. When a new process picks up the run, it replays the history to rebuild state, and steps that already completed return their recorded results instead of executing again.
- Isolate side effects. Anything that touches the outside world runs as a discrete, recorded step, so “already done” is a fact the system knows rather than something the model has to guess.
- Keep orchestration deterministic. The code that decides what runs next must produce the same decisions on replay. I/O belongs in the recorded steps, not in the orchestrator.
- Retry at the step level. If the supplier email API times out, retry that one call with backoff. Do not rerun the whole agent.
With this model, an instance being recycled stops being an incident. Another worker picks up the run, replays to the point of failure, and continues.
One caveat matters: replay protects you from repeating completed steps, but a step that crashed midway may still run again. Design tools that write to external systems to be idempotent where you can, for example by passing a stable request key derived from the run and order ID.
Implementing it with ADK and Dapr Workflow
Dapr is an open-source CNCF graduated project that runs as a sidecar next to your application. Its workflow engine keeps an event-sourced history in a state store you choose, such as PostgreSQL or Redis, replays after a crash, and does not re-execute completed activities. It also supports step retries with exponential backoff and durable timers.
You could wrap an ADK agent in Dapr Workflow by hand, but you would be rebuilding the agent loop as an orchestrator. Diagrid, a company founded by the creators of Dapr, publishes an integration in the diagrid Python package on PyPI that does this for you. Its DaprWorkflowAgentRunner for ADK runs the agent loop as a workflow, and each tool execution becomes a durable Dapr activity.
The agent definition stays standard ADK:
from google.adk.agents import LlmAgent
from google.adk.tools import FunctionTool
from diagrid.agent.adk import DaprWorkflowAgentRunner
agent = LlmAgent(name=”assistant”, model=”gemini-2.5-flash”, instruction=”…”, tools=[FunctionTool(get_order)])
runner = DaprWorkflowAgentRunner(agent=agent, name=”order-agent”, max_iterations=10)
runner.start()
async for event in runner.run_async(user_message=”…”, session_id=”s-001″, user_id=”u-42″, app_name=”orders”): …
For the supply-chain agent, get_order, the inventory reallocation function, and the supplier email function would each be registered as tools. When the instance disappears partway through the fourth email, the run resumes on another replica. The order lookups, the 18 reallocations, and the three sent emails are already recorded, so the agent picks up with email four. Tool results are recorded; verify how model turns behave on resume in your own tests. The max_iterations setting also caps runaway loops.
Install the extra with pip install “diagrid[adk]”. Locally, you can run it with dapr init and dapr run –app-id <id> –resources-path ./resources — python3 agent.py, with a state store component that sets actorStateStore: “true”. A good first test is to kill the process mid-run and watch it pick up where it stopped. Test any durable execution for AI agents setup against your own tools and failure modes before trusting it in production.
Where to run it on Google Cloud
The Dapr sidecar and its control plane are designed for Kubernetes, VMs, and self-hosted environments. On Google Cloud, that makes GKE the natural home for long-running durable agents. You install Dapr into the cluster, point the workflow state store at something like Cloud SQL for PostgreSQL or Memorystore for Redis, and let your agent pods be drained, rescheduled, or preempted without losing runs. Spot node pools become a reasonable cost option once a reclaimed node means a short delay rather than a broken run.
Cloud Run still has a role. It is a good fit for the stateless parts: the API that accepts a request, validates it, starts an agent run, and returns a run ID, or a small UI that shows run status. Keep the long-lived agent execution on GKE or another runtime where the durable workflow engine runs alongside it. If you want to run Dapr-based components on Cloud Run specifically, check current documentation and test it yourself rather than assuming it works.
If you would rather not operate Dapr and its state stores, a managed durable runtime is the other route. Catalyst from Diagrid is one option: it is built on Dapr, supports the ADK integration among others, and can be Diagrid-hosted, including on a dedicated cluster on Google Cloud, or run in your own cloud or on-premises. As with any vendor, compare it against running open-source Dapr yourself, and validate recovery behavior under realistic failure tests.
A rollout checklist
- ☐ List every tool your agent calls and mark which ones have external side effects.
- ☐ Make side-effecting tools idempotent using a stable key per run and per business entity.
- ☐ Choose a workflow state store you already know how to back up, monitor, and secure.
- ☐ Move long-running agent execution to GKE (or a managed durable runtime), and keep Cloud Run for stateless entry points.
- ☐ Wrap your ADK agent with a durable runner so each tool execution is a recorded activity.
- ☐ Set max_iterations to a value that fits the task, not an arbitrary large number.
- ☐ Kill a pod mid-run in a staging cluster and confirm no tool runs twice.
- ☐ Drain a node and run a GKE upgrade during an active run as part of your test plan.
Conclusion
Cloud Run and GKE will keep recycling instances, draining nodes, and reclaiming spot capacity. Fighting that with longer timeouts only delays the next failure. The durable fix is to give the agent loop a persistent history, so a restart becomes a resume and every completed tool call stays done. Start with one agent that has real side effects, break it on purpose in staging, and measure whether it recovers cleanly before you scale the pattern out.