Callbacks

You can implement the following callbacks in your Interactive Canvas Action:

onUpdate()

The onUpdate()callback passes data from your webhook to your web app to update the web app appropriately. You should only use this callback with the server-side fulfillment model of Interactive Canvas development.

For more information about onUpdate(), see Pass data to update the web app.

onTtsMark()

The onTtsMark() callback is called when custom <mark> tags included in the Speech Synthesis Markup Language (SSML) of your response are read out to the user during Text to Speech (TTS). You can use onTtsMark() in both the server-side and client-side fulfillment development models.

In the following snippets, onTtsMark() synchronizes the web app's animation with the corresponding TTS output. When the Action has said to the user, "Sorry, you lost," the web app spells out the correct word and displays the letters to the user.

In the following example, the webhook handler revealWord includes a custom mark in the response to the user when they've lost the game:

JavaScript

…
app.handle('revealWord', conv => {
  conv.add(new Simple(`<speak>Sorry, you lost.<mark name="REVEAL_WORD"/> The word is ${conv.session.params.word}.</speak>`));
  conv.add(new Canvas());
});
…

The following code snippet then registers the onTtsMark() callback, checks the name of the mark, and executes the revealCorrectWord() function, which updates the web app:

JavaScript

…
setCallbacks() {
    // declare Assistant Canvas Action callbacks
    const callbacks = {
        onTtsMark(markName) {
            if (markName === 'REVEAL_WORD') {
                // display the correct word to the user
                that.revealCorrectWord();
            }
        },
    }
    callbacks.onUpdate.bind(this);
}
…

onInputStatusChanged()

The onInputStatusChanged() callback notifies you when the input status changes in your Interactive Canvas Action. Input status changes indicate when the microphone opens and closes or when Assistant is processing a query. The following events can cause the input status to change:

  • The user speaking to your Action
  • The user inputting text on the Android Google Search App (AGSA)
  • The web app using the sendTextQuery() API to send a text query to the Action
  • The Action writing to home storage and other Assistant events

The primary use case for this callback is synchronizing your Action with the user’s voice interactions. For example, if a user is playing an Interactive Canvas game and opens the microphone, you can pause the game while the user speaks. You can also wait until the microphone is open to send a text query to Assistant to ensure it's received.

This API reports the following statuses:

  • LISTENING - Indicates that the microphone is open.
  • IDLE - Indicates that the microphone is closed.
  • PROCESSING - Indicates that Assistant is currently executing a query, and the microphone is closed.

The API reports the input status to your Action each time the status changes.

While any transition between states is possible, the following flows are common:

  • IDLE>LISTENING>PROCESSING>IDLE - The user says a query, the query is processed, and the microphone closes.
  • IDLE>PROCESSING>IDLE - The web app uses the sendTextQuery() API to send a text query to the Action.
  • IDLE>LISTENING>IDLE - The user opens the microphone but does not say a query.

To use this feature in your Action, add onInputStatusChanged() to your web app code, as shown in the following snippet:

onInputStatusChanged(inputStatus) {
   console.log("The new input status is: ", inputStatus);
}

The onInputStatusChanged() callback passes back a single enum parameter, inputStatus. You can check this value to see the current input status. The inputStatus can be LISTENING, PROCESSING, or IDLE.

Next, add onInputStatusChanged() to the callbacks object to register it, as shown in the following snippet:

 /**
  * Register all callbacks used by the Interactive Canvas Action
  * executed during game creation time.
  */
 setCallbacks() {
   const that = this;
   // Declare the Interactive Canvas action callbacks.
   const callbacks = {
     onUpdate(data) {
       console.log('Received data', data);
     },
     onInputStatusChanged(inputStatus) {
       console.log("The new input status is: ", inputStatus);
     },
   };
   // Called by the Interactive Canvas web app once web app has loaded to
   // register callbacks.
   this.canvas.ready(callbacks);
 }
}