Capabilities and examples
These examples follow a robot support assistant. A saved Template determines enabled tools and dependencies. Each Session chooses its model independently.
Conversation and streaming
Section titled “Conversation and streaming”An End User asks: “Help me diagnose this robot.” Submit the message through POST /api/v1/user/sessions/{id}/events and keep the JSON command receipt. Your backend follows progress and answers over its Backend Connection, or reads the receipt and the Turn.
A new plain-text requirement during execution can steer the same Turn. An uncertain submission is recovered through its original receipt rather than a new execution. Disconnecting does not stop accepted work; cancellation is an explicit command. See Session events and recovery.
Realtime voice
Section titled “Realtime voice”An End User says: “Explain this robot’s temperature readings.” Open a Realtime Connection on their existing Session. The Realtime Model handles speech; when it delegates analysis, the Full Agent runs a task using that Session’s fixed configuration and enabled tools. Your application can play the reply, show transcripts, interrupt playback and display task progress linked to its Turn.
A task acknowledgement is separate from a saved task result. Realtime chat does not create a Full Agent Turn. Read the Realtime voice guide for connection tokens, PCM16/Opus audio, WebSocket events and current recovery limits.
Tools and Sandbox
Section titled “Tools and Sandbox”Ask: “Calculate the average of these temperatures: 31, 33 and 32 °C.” With Sandbox enabled, the Agent can use its execution tools to calculate 32 °C and explain the result. The Template and Sandbox profile determine available tools; the request does not force a particular tool choice.
Files and network access follow enabled capabilities and policy. Already completed external effects keep their actual effects after steering or stopping. Architecture shows the controlled execution boundary.
Upload a measurement file through /api/v1/user/drive/uploads and complete its size and digest verification. Select the ready file in a message and ask: “Explain the anomalies.” The enabled file tools make its private content available to the Agent; an upload intent alone does not make content ready.
Browse and read through /api/v1/user/drive/entries and /drive/files/content. Short-lived download tickets provide a separate authorized download flow. See Drive files.
Devices
Section titled “Devices”Ask: “Is my robot charging?” Your application declares the robot’s actions as Client Tools on the End User’s Session. The Agent calls one, your application claims the call over its Backend Connection, forwards it to the robot and submits the result, and the model answers from it.
Your application or device can declare Client Tools on an ordinary text Session and execute their calls. The platform routes required actions and results; your integration owns device transport and physical execution. Voice device actions delegate to Full Agent Turns, which use the same Session Client Tools; Realtime exposes no direct device or Session-client tools.
Usage and billing
Section titled “Usage and billing”Two End Users share one support Template. Usage is attributed to their Organization and each End User, including work your backend authorizes with an Organization Key. Those requests retain the actual Organization-Key identity. Your Organization backend reads a target’s summary at GET /api/v1/org/users/{userId}/billing/usage and paginated details at /billing/usage/details.
The platform sets no spending limit before work starts: usage is priced at the platform sale price in the deployment’s billing currency and reported afterwards. Reporting covers LLM, Sandbox and Web Search usage. Usage on customer-provided model credentials is reported without a platform price.
Continue with Quickstart or the API reference.