Skip to main content

Automation and Agents

Gemini 3.8 Live: Keep Async Tool Results on Track

A voice conversation can continue while a lookup runs. The engineering job is keeping its result attached to the right request when the caller changes direction.

A voice waveform stays connected to its request tag beneath the Gemini family mark.
On this page
  1. Choose the exact Live model contract
  2. Follow the message route, not a guessed socket format
  3. Give each lookup a durable owner
  4. Return the result with the documented shape
  5. Separate playback interruption from tool cancellation
  6. Store completion where the business action happens
  7. Reconnect the session without orphaning the work
  8. Test the state transitions before adding more tools

A caller asks for an appointment at the downtown office. While the lookup runs, they change their mind: the north office would be easier. A voice agent can keep the conversation moving, but the first result still needs an owner. Otherwise, an answer about downtown can arrive inside a conversation that has already moved north.

Gemini 3.8 Live supports asynchronous function calling. That makes this a useful engineering problem to solve: keep listening, handle completed work, and preserve which request each result belongs to. A smooth spoken response cannot substitute for that state.

This guide develops a hypothetical appointment lookup using Google's current documentation, a small JavaScript response fragment and a proposed testing procedure. It is a source-based design, not a report of an executed Gemini integration or measured latency. Start with a read-only lookup so that mistakes can be examined without creating appointments.

Key takeaways

  • Keep the model's function-call ID and your application's business-request identity together.
  • Treat audio interruption, tool cancellation and external completion as separate events.
  • Test result order, changed requests and reconnects before adding actions that write to another system.

Choose the exact Live model contract

For standard gemini-3.8-live, Google's model migration guidance identifies non-blocking execution as the default. A declaration can explicitly choose NON_BLOCKING or select BLOCKING for compatibility. The same guidance says to omit thinking_level or thinking_config for this model. Record those details in the setup you review. An older sample can carry defaults or settings for a different model. Gemini 3.8 Live documentation

Gemini 3.8 Live Extended Thinking is a separate choice. Google's capabilities comparison describes async-only tools, no function scheduling, and an interaction lifecycle in which turnComplete alone does not establish an idle session. An integration designed for the standard model should not switch variants and assume its state transitions remain appropriate. Live API capabilities guide

There is documentation drift to watch for. The tools guide still includes an older model-support table while its examples use 3.8. Read the exact model page with the current capability comparison. Record the model string and SDK version in your integration notes so a later engineer can reproduce the choices instead of guessing which example supplied them.

An explicit declaration also makes review easier. In our proposed lookup, the name might be lookup_open_slots, with the office and date supplied as arguments. Its description should say that it reads availability and does not reserve a slot. In the application contract, also define how office identity, date and timezone are represented. An ambiguous local date can return technically valid availability for the wrong day. Keep that interpretation in application code instead of relying on an improvised spoken explanation. That distinction lets the surrounding application explain what has happened accurately: searching is underway, availability was returned, or a lookup failed. None means an appointment exists.

Make the tool small enough to understand. A single declaration that searches, chooses a time, books it and sends a confirmation hides several different outcomes behind one success value. Separate those responsibilities before deciding which can run while conversation continues. The first implementation should make one office lookup understandable from start to finish. That gives the team a concrete trace to inspect before a reservation write is introduced.

A magnifying lens examines a calendar beside a separate blank appointment card. View image detail

Choose Actual size to read the graphic closely.

Follow the message route, not a guessed socket format

A Live session carries several kinds of events over one connection. Google's WebSocket reference describes JSON client messages for setup, client content, realtime input and tool responses. The server can send content, tool calls, cancellation notices and session information. A client should inspect those message types, rather than assume that everything received over a socket is playable audio. Live API WebSocket reference

Prefer the documented SDK connection flow unless the project needs a raw protocol implementation. A socket path invented from a model name is not a substitute for the service reference. The model belongs in configuration; it does not establish that a particular handwritten endpoint or authentication arrangement is valid.

The audio format also needs an explicit boundary. Google's guide specifies little-endian, 16-bit PCM; input is natively 16 kHz, with resampling available when the declared rate differs, and output is 24 kHz. Its JavaScript example puts base64 audio in a Blob-shaped input with a MIME type containing the sample rate. Those are media contents inside the supported message route, not instructions to forward every socket buffer as a binary PCM frame. Audio formats and sending audio

Give audio handling its own adapter. Capture, conversion, message parsing and playback are separate steps. A microphone can produce data in a format different from what the model expects. An output event can contain more than one content part. Handle relevant parts deliberately and give malformed or unsupported media a defined failure path.

For the appointment example, a tool-call event starts application work while the audio adapter continues handling the session. The lookup runner should not occupy the receive loop until its network request finishes. The event receiver should remain able to process new speech, interruption and connection notices. This is a proposed application architecture, not a claim about hidden model inference tracks.

A microphone feeds an audio-message envelope, which routes toward a speaker. View image detail

Choose Actual size to read the graphic closely.

Give each lookup a durable owner

Store two identities: the model's call ID and an application request ID. The call ID associates a function response with its function call. The request ID identifies the business question you are trying to answer. A second lookup can belong to a revised version of that question even when it arrives in the same voice session.

In the hypothetical appointment flow, store the caller's chosen office and date with a request version. When downtown changes to north, mark the downtown lookup as superseded and create or update the north request according to your application's rules. When either lookup finishes, compare its recorded scope with the currently relevant request before using its result in an answer.

A clarification is different from a replacement. If the caller asks whether the office has parking, the original availability lookup may still be useful. Advancing a global counter on every utterance and rejecting everything with an older number would discard valid work. Make supersession an explicit business decision tied to the changed fields, not an automatic reaction to any new speech.

A proposed job record can hold the call ID, function name, request ID, request version, office, date, start time and status. Use status names your application defines, such as pending, completed, failed or superseded. These are application fields, not additional Gemini protocol fields. They make a late result explainable without asking the model to infer which request it once belonged to.

This same ownership question appears in Rise's examination of embedded app approval continuations. A message reaching a visible chat does not establish that it reached the action waiting for it. Here, a result reaching the voice session does not establish that it answers the caller's current question. Follow the specific pending work through the handoff.

Do not put the entire request ledger into every spoken response. The caller needs a useful answer about the right office. The engineer needs enough protected state to trace why that answer was selected. Good internal records let the product remain simple while its behavior stays understandable.

Each of two offices connects to its own request tag and result folder. View image detail

Choose Actual size to read the graphic closely.

Return the result with the documented shape

The client executes a tool and sends its response. Google's tools guide says Live API tool responses must be handled manually; automatic function-response handling is not part of that Live flow. Preserve the received call's ID and name when constructing the reply. Tool use with Live API

The following JavaScript fragment illustrates only the response boundary. session is an already connected SDK session. call is a received function call that the surrounding dispatcher has validated. output is the application's validated lookup result. The fragment does not connect a microphone, authenticate a user, execute a lookup or constitute a complete server.

javascript
import { FunctionResponseScheduling } from '@google/genai';

function returnLookupResult(session, call, output) {
  session.sendToolResponse({
    functionResponses: [{
      id: call.id,
      name: call.name,
      response: { output },
      scheduling: FunctionResponseScheduling.WHEN_IDLE
    }]
  });
}

The current SDK reference places scheduling alongside response in FunctionResponse. That differs from an example in the general tools guide that nests scheduling inside the result object. Use the SDK's documented type shape for this fragment and check it against the version installed in your project. Its WHEN_IDLE default concerns how a non-blocking response is scheduled; it does not validate the appointment data. FunctionResponse SDK reference

The dispatcher still has work to do before calling this function. Allow only the tools your application actually exposes. Validate the office and date, apply the caller's existing access rules, and validate the service's response. A schema in a declaration helps describe arguments, but it should not replace the application's checks or imply a guarantee that every model-supplied value is usable.

Send a bounded result that supports the answer. In this example, return the office and date searched, available times and a clearly defined status. If the upstream lookup fails, preserve a defined error outcome instead of manufacturing an empty success. Define the result's business meaning so the agent can distinguish a failed search from a successful search with no available times. Those are different answers for a caller deciding whether to try another office.

Two hands pass a result folder from one side to the other. View image detail

Choose Actual size to read the graphic closely.

Separate playback interruption from tool cancellation

When the server reports serverContent.interrupted, the playback adapter should stop current output and clear queued audio. That flag is server output, not a fabricated clientContent.interrupted input. Google's capabilities guide also documents automatic voice activity detection by default; a client taking over detection must explicitly choose that mode and send the corresponding activity events. Voice activity detection and interruptions

That handles the audible response. It does not, by itself, establish that a lookup stopped or that the caller withdrew every pending request. In the appointment example, “Wait, I meant north” changes the location. “Wait, can you repeat the date?” may interrupt speech while leaving the lookup relevant. Tie application decisions to the meaning of the correction and the current request record.

The protocol separately defines toolCallCancellation with specific call IDs. Process a received cancellation against those calls, rather than treating it as a universal cancel instruction. Google's reference also distinguishes cancellation from reversing side effects. A client may need to attempt an undo where an external operation already did something. Do not promise that an audio event automatically prevents a remote action. Tool-call cancellation reference

An application can use an AbortController for a supported in-flight fetch or stream. MDN's AbortController reference describes that local capability. It is useful for a superseded read-only lookup, but your business service must establish what happened if an operation could already have completed. Stopping the local wait cannot prove a remote booking was undone.

Make that distinction visible in your implementation logs. Record playback interruption, application supersession, a received cancellation and the service's eventual outcome separately. If a cancellation notification never arrives in a particular scenario, the application still needs a rule for changed requests. Test the actual behavior instead of assuming that every interruption will produce the same sequence of events.

Stopped audio playback sits apart from a work folder marked as pending. View image detail

Choose Actual size to read the graphic closely.

Store completion where the business action happens

Once you add reservation writes, the appointment system's record becomes essential. The model's spoken confirmation is a presentation of an outcome, not the outcome itself. A job might still be pending, a booking might have succeeded before a connection failed, or an old availability result might no longer be usable.

Give the write a business operation identity that survives its conversational presentation. Keep the requested office, slot and actor with the operation, and decide where a supported retry can recover its status. If your booking service provides an idempotency mechanism, use its actual contract. Do not assume a Gemini call ID automatically becomes the external service's duplicate-prevention key.

AWS's original Builders' Library explanation describes using a caller-provided unique request identifier to recognize retries, and the importance of recording that identity atomically with the operation. That is independent engineering context for the proposed reservation design, not evidence of Gemini performance. Making retries safe with idempotent APIs

Suppose the appointment service commits the booking but the response connection drops. The application should classify the outcome as uncertain until it can inspect the service's supported status or retry behavior. Immediately submitting another booking with a new identity could create a second operation. Saying “nothing was booked” would also outrun the available evidence.

The spoken update should reflect that uncertainty: the application is checking whether the reservation completed. Once the service confirms the actual outcome, present that outcome with the correct request. If it cannot be established, route the case according to your product's defined recovery path. The recovery record should preserve the original operation identity and the last confirmed service state. That gives the next person handling the case a concrete starting point.

A returning operation tag points toward its ledger entry while a lens examines the blank outcome page. View image detail

Choose Actual size to read the graphic closely.

Reconnect the session without orphaning the work

A connection and a business job have different lifetimes. Google's session-management guide documents session resumption updates, resumption handles and GoAway notices. Resumption supports the conversation connection; it is not your appointment database or a record of external commits. Session management with Live API

Persist enough application state to find the pending lookup after a reconnect. On resumption, reconcile the relevant jobs with the current request before deciding what to send or say. A completed downtown result should not become north availability just because both existed before the socket disconnected.

Choose credential design separately from that job ledger. A server-mediated architecture can keep long-lived credentials on the backend. Direct client access is also a documented option: Google's ephemeral-token guide describes a backend authenticating the client and issuing a short-lived, restrictable token for the Live API. That corrects the blanket assumption that every browser must proxy all realtime media through your server. Ephemeral tokens

Choose the route according to the application's media, access and operational requirements. If tools need trusted backend data, keep those service calls behind the appropriate application boundary regardless of where the media connection terminates. The model token and the appointment service's access rules serve separate purposes. Evaluate each at its own boundary.

During recovery, distinguish reconnecting the model session from retrying external work. After the socket opens, inspect the current job status before starting another lookup or write. Connection recovery and business recovery need separate evidence. Store sensitive tokens and customer data appropriately; public traces can show matched identities and status transitions without exposing their values.

Separated connection plugs sit above a retained work folder and hourglass. View image detail

Choose Actual size to read the graphic closely.

Test the state transitions before adding more tools

Use a controlled read-only service whose results and completion order you can choose. This proposed experiment tests your integration's state behavior; it has not been performed for this article. Begin with one office lookup, then introduce a caller correction while that lookup remains pending. Complete the original and revised jobs in both orders.

For each run, record the model and SDK version, setup choices, received tool-call identity, application request version and the actual events observed. Record when the lookup starts and finishes separately from when audio reaches the player. That separation helps diagnose whether a delay came from the service, adapter, model interaction or playback, without declaring one universal voice latency target.

Add a clarification that should preserve the lookup, such as asking about parking. Then add a correction that should supersede it. The acceptance condition is specific: the application keeps relevant work, identifies superseded work and never presents downtown availability as the north result. A general “the conversation sounded natural” score would miss that failure.

Interrupt playback with a known pending call and inspect the resulting sequence. Confirm that the queue clears. Separately record whether a cancellation notice arrives, what the application marks superseded and whether the service completes anyway. Run a reconnect with a pending lookup and inspect whether the same business request remains identifiable afterward.

Finally, test a defined failure and a duplicate delivery or retry through the application's supported path. Leave observed results blank until each run finishes. Preserve failures rather than filling a checklist from expected behavior. Then examine a complete conversation for misleading statements: a lookup acknowledgment should not sound like a reservation confirmation.

The first useful milestone is small: one documented read-only tool, one request ledger and observed handling of a changed request. Once those results are clear, add the next tool with the same ownership rules. Async voice earns its place when the caller can change direction and the application still delivers the right result.

Two differently tagged result folders approach a blank observation clipboard. View image detail

Choose Actual size to read the graphic closely.

Checked for this article

Sources

  1. Gemini 3.8 Live model documentationGoogle
  2. Live API capabilitiesGoogle
  3. Live API toolsGoogle
  4. Live API protocol referenceGoogle
  5. JavaScript SDK FunctionResponseGoogle
  6. Live API ephemeral tokensGoogle
  7. Live API session managementGoogle
  8. Making retries safe with idempotent APIsAmazon Web Services
  9. AbortController referenceMDN
  10. Agent Native app approval and resumeRise Productive
  11. Gemini 3.8 Live and Extended Thinking announcementGoogle

Keep reading

All articles