Events

Overview

Register handlers with experience.on(event, handler) after Experience.startSession(). Component integrations use the matching callback props (onMessage, onActivityChange, …).

Startup

ready means the player visuals are mounted. started means the player-owned start button was used and audio is unlocked.

const experience = await Experience.startSession({
experienceId: 'exp_…'
});

experience.on('ready', (evt) => {
// evt: ExperienceEventEnvelope<{ session }>
console.log('Player visuals ready', evt.data.session.id);
});

experience.on('started', (evt) => {
// evt.data.mode — experience mode after audio unlock
console.log('Audio and session started in', evt.data.mode, 'mode');
});

await experience.attach({ container });

ready includes the public session facts (not a parseable launch). started includes the experience mode. Both replay asynchronously for handlers registered after the event. The onStart option on Experience.startSession() is a convenience callback for startup completion.

Speech and transcripts

Emitted during speak(), listening, and custom processor conversation. Partial transcript revisions include monotonic revision and optional delta. Only updates with isFinal: true commit a user message to conversation history.

experience.on('userTranscript', (evt) => {
  const update = evt.data;
  if (!update.isFinal) {
    liveCaptionEl.textContent = update.text;
    return;
  }
  liveCaptionEl.textContent = update.text;
  commitUtterance(update.utteranceId, update.text);
});

experience.on('userSpeechStarted', () => micIndicator.classList.add('active'));
experience.on('userSpeechEnded', () => micIndicator.classList.remove('active'));

experience.on('characterSpeechStarted', (evt) => {
console.log('Speaking', evt.data.characterId, evt.data.text, evt.data.source);
});

experience.on('characterSpeechEnded', (evt) => {
console.log('Speech ended', evt.data.reason, evt.data.durationMs);
});

experience.on('conversationUpdate', (evt) => {
console.log('History length', evt.data.length);
});

experience.on('listeningState', (evt) => {
console.log('Mic gate', evt.data);
});

Custom conversation processor

When you supply conversationProcessor on startSession(), processor failures surface as conversationProcessorError. The managed LLM is not used as a fallback.

experience.on('conversationProcessorError', (evt) => {
console.error('Processor failed for', evt.data.utteranceId, evt.data.message);
});

Conversation

// Every experience.on handler receives an envelope:
// { id: 'evt_…', type, sessionId, timestamp, data }

experience.on('message', (evt) => {
// evt.data.status is always 'final' for committed history messages
console.log(evt.data.role, evt.data.text, evt.data.source, evt.data.status);
});

experience.on('activityChange', (evt) => {
// evt.data: 'idle' | 'listening' | 'thinking' | 'speaking'
console.log('Activity', evt.data);
});

Lip-sync and facial animation run inside the hosted player. Integrators do not receive viseme or animation keyframe events — use characterSpeechStarted / characterSpeechEnded to sync host UI with speech.

Player attach callbacks

Embed lifecycle is on the object returned from attach() (and on component props), not on experience.on():

const player = await experience.attach({
container,
onPlayerStatusChange: (status) => {
  // PlayerStatus: loading | scene | ready | error
  console.log(status);
}
});

player.on('close', ({ reason, returnUrl }) => {
console.log('Player closed', reason, returnUrl);
});

Event reference

Event	Payload	Description
ready	envelope .data: { session }	Player visuals are mounted and ready
started	envelope .data: { mode }	Player startup click, audio unlock, and session startup completed
userTranscript	envelope .data: { utteranceId, text, revision, isFinal, delta? }	Partial and final STT updates. Ephemeral until isFinal: true.
userSpeechStarted	envelope .data: { utteranceId? }	VAD detected speech activity (auto mode telemetry)
userSpeechEnded	envelope .data: { utteranceId? }	VAD detected end of speech; final text may arrive after this event
characterSpeechStarted	envelope .data: { speechId, turnId, characterId, text, source }	Character began speaking (speak, opening, or llm)
characterSpeechEnded	envelope .data: { speechId, turnId, characterId, text, source, durationMs?, reason }	Character speech finished or was interrupted
characterFocusChanged	envelope .data: { characterId, reason }	Presentation focus changed on a multi-Character node (reason: speech | api | node_enter). Not emitted on single-Character nodes.
conversationUpdate	envelope .data: ConversationMessage[]	Immutable snapshot of in-session conversation history
listeningState	envelope .data: boolean	Manual listening gate opened (true) or closed (false)
message	envelope .data: ConversationMessage	Durable user or assistant message (status: ‘final’, plus source, role, text, ids)
activityChange	envelope .data: ‘idle’ | ‘listening’ | ‘speaking’ | ‘thinking’	Session listening / speaking activity for UI sync
interactionChange	envelope .data: ExperienceInteraction	Captions, text/mic allow + enabled state, mute character speech, or Edit (allowRewind) changed — from the player or after setInteraction()
feedback	envelope .data: { contractVersion: 'feedback.v3', types: [...] }	End-of-session Feedback (when Feedback is attached and scoring succeeds). Always emitted when available — even if the player also shows Feedback UI. Each types[] entry is discriminated by kind: score, outcome, or rubric. Prefer result.presentation.visibleOverall (score/rubric) or result.label (outcome) for display; do not invent totals from outcome results.
feedbackFailed	envelope .data: { error, model? }	End-of-session Feedback scoring failed
conversationProcessorError	envelope .data: { utteranceId, message }	Browser conversationProcessor threw or rejected

Feedback result shapes (feedback.v3)

Studio authors choose one Feedback kind per library item. The player event carries that kind on each entry — handle with a switch on kind.

experience.on('feedback', (evt) => {
for (const item of evt.data.types) {
  switch (item.kind) {
    case 'score':
      // One overall score + optional grade / written message
      console.log(item.result.presentation.visibleOverall, item.message);
      break;
    case 'outcome':
      // Succeeded / Failed (or custom labels) — not a numeric score
      console.log(item.result.label, item.result.value, item.message);
      break;
    case 'rubric':
      // Per-criterion scores + overall presentation
      console.log(item.result.presentation.visibleOverall, item.result.criteria);
      break;
  }
}
});

Not on experience.on()

Signal	How to listen
Player embed state	attach({ onPlayerStatusChange }) or component onPlayerStatusChange
Player close	player.on(‘close’, …) after attach(), or component onClose