Experience API

Overview

Experience is the primary JavaScript API. Use it when you need programmatic control beyond the component — custom containers, event handling, presenter sequencing, or browser-owned conversation logic.

Start a session

import { Experience } from '@liforma/client';

const experience = await Experience.startSession({
  experienceId: 'exp_T0I7ACMQLBMPG6K'
});

await experience.attach({ container: '#avatar' });

Session inputs

Browser mint configuration

Pass these options to Experience.startSession(). They are forwarded when the SDK mints via the browser endpoint.

Conversation sessions default to conversation, managed responses, and auto input. Presenter sessions default to presenter, manual responses, and manual input.

Supply conversationProcessor for browser-owned responses. Registered server processors (opaque processorId) are a separate upcoming capability.

Player-owned startup

The embedded player always renders and owns the actual startup click. Browsers require user activation inside the cross-origin player frame before it can unlock audio, so a button in the host page cannot replace this interaction.

Use the startButton property on attach() (or the matching <Experience> prop) to adapt the built-in control to your interface. Supported placement values are center, bottom-center, bottom-left, and bottom-right. Supported variants are primary, secondary, and minimal. Use theme for shared chrome (introduction, feedback, panels); startButton.appearance still wins on the start CTA.

Pass interaction on attach() (or the matching <Experience> / widget option) to merge defaults and control visibility before first paint. Nested controls patches are deep-merged. This does not remint.

Top-left player chrome uses the same visibility props as closeButton: settingsButton (default on), theatreButton (default off), and fullscreenButton (unset = size heuristic). Composer buttons stay on interaction.controls.

const experience = await Experience.startSession({
  experienceId: 'exp_T0I7ACMQLBMPG6K',
  mode: 'presenter'
});

experience.on('started', async (evt) => {
  if (evt.data.mode === 'presenter') {
    await experience.speech.speak({ text: 'Welcome to the lesson.' });
  }
});

await experience.attach({
  container: document.querySelector('#avatar'),
  theme: {
    colors: { primary: '#635bff', primaryText: '#ffffff' },
    radiusPx: 16
  },
  startButton: {
    label: 'Begin lesson',
    ariaLabel: 'Begin the guided lesson',
    placement: 'bottom-center', // center | bottom-center | bottom-left | bottom-right
    variant: 'primary', // primary | secondary | minimal
    appearance: {
      backgroundColor: '#635bff',
      textColor: '#ffffff',
      borderColor: '#8179ff',
      borderRadiusPx: 999,
      borderWidthPx: 2,
      size: 'large',
      fontFamily: 'system-ui, -apple-system, sans-serif',
      fontWeight: 700,
      // Or set an exact label size (10–48); overrides size font scaling:
      // fontSizePx: 22,
      shadow: 'soft'
    },
    icon: 'player-play', // Tabler id, or https://… / data:image/…
    iconPosition: 'left' // left | above
  }
});

started fires after the player button is clicked and audio is unlocked. In presenter mode, it is the safe point for host code to call speak(). Calling speak(), startListening(), stopListening(), or listenOnce() before started throws a clear error. Button appearance is limited to the documented tokens; arbitrary CSS is not supported across the player frame. For a large pill button use size: 'large' with borderRadiusPx: 999, or set fontSizePx (10–48) for an exact label size. fontFamily accepts system / stack names available in the player iframe (host webfonts are not loaded there); fontWeight is 100–900. Optional icon accepts a Tabler Iconify id (e.g. player-play) or an absolute https://… / data:image/… URL; iconPosition is left (default) or above. Tabler icons inherit the button text colour. Site-relative paths are not supported (they would resolve against the player iframe origin).

Speech API

Call attach() first, wait for started, then use the methods below. See Events for characterSpeechStarted, userTranscript, userSpeechStarted, and related handlers.

Prerequisites

Speech methods throw if the session has not started. Register experience.on('started', …) (or the onStart option on startSession) and only call speech methods after the player-owned start button unlocks audio.

speech.speak()

Queue animated character speech via Liforma TTS without invoking the managed LLM. Requires textSpeech. Resolves when playback completes.

const result = await experience.speech.speak({
  text: 'Repeat after me: Buenos días.',
  characterId: 'char_…', // optional — defaults to activeCharacterId
  queue: 'append' // optional — append (default), replace-active, or replace-all
});

console.log(result.utteranceId, result.durationMs, result.status);

speech.play() and createUtterance()

Bring-your-own audio. Requires externalSpeechAudio. Pass PCM, encoded bytes (audio/mpeg, …), or a CORS-open URL. Encoded/URL sources are decoded in the player, then lipsynced via speech-animation windows. See Bring your own voice.

When you have the spoken text, pass optional transcript on play, createUtterance, utterance.setTranscript(text), or close({ transcript }). Audio alone is enough; providing the text when available usually improves lipsync. Empty / whitespace is ignored.

// Encoded bytes — decoded in the player (not on api.liforma.ai)
await experience.speech.play({
  audio: { data: mp3Bytes, encoding: 'audio/mpeg' },
  queue: 'append'
});

// Or a CORS-open URL fetched + decoded inside the player iframe
await experience.speech.play({
  audio: { url: 'https://cdn.example.com/line.mp3' },
  queue: 'append'
});

type TurnState = {
  utterance: ReturnType<typeof experience.speech.createUtterance>;
  writes: Promise<void>;
};

const turns = new Map<string, TurnState>();

function beginTurn(turnId, transcript) {
  const utterance = experience.speech.createUtterance({
    format: { encoding: 'pcm_s16le', sampleRate: 24_000, channels: 1 },
    queue: 'replace-active',
    // Optional seed — pass spoken text when available (helps lipsync)
    ...(transcript ? { transcript } : {})
  });
  turns.set(turnId, { utterance, writes: Promise.resolve() });
}

function writeTurn(turnId, chunk) {
  const turn = turns.get(turnId);
  if (!turn) return;
  const u = turn.utterance;
  turn.writes = turn.writes
    .then(() => u.write(chunk))
    .catch((error) => {
      console.error('Unable to write speech audio', error);
      void u.cancel();
    });
}

async function endTurn(turnId, transcript) {
  const turn = turns.get(turnId);
  if (!turn) return;
  turns.delete(turnId);
  await turn.writes;
  await turn.utterance.close({ transcript, history: 'none' });
}

async function cancelTurn(turnId) {
  const turn = turns.get(turnId);
  turns.delete(turnId);
  if (turn) await turn.utterance.cancel();
}

speech.interrupt()

Returns settled SpeechResult[] for interrupted/cancelled utterances. Options: { scope: 'active' | 'all' }, { utteranceId }, or { characterId }.

startListening() and stopListening()

Open and close a logical microphone gate. The player acquires the physical microphone once during startup; stopListening() finalizes the utterance without stopping the media track. Use with speechInputMode: 'manual' so learner pauses do not end the utterance.

await experience.startListening();
// Learner speaks; pauses do not end the utterance in manual mode.
const utterance = await experience.stopListening();
console.log(utterance.utteranceId, utterance.text);

stopListening() returns UtteranceResult with utteranceId and text. In managed conversation mode, a finalized manual utterance still triggers the configured processor. In presenter mode, the host decides what to do with the text.

listenOnce()

Convenience for automatic end-of-speech capture. Requires speechInputMode: 'auto'. Opens one listening gate, waits for VAD to finalize a single utterance, then resolves with UtteranceResult. Use for quiz answers, voice forms, or presenter flows where the host owns the turn loop but does not want Start/Stop buttons.

const experience = await Experience.startSession({
  experienceId: 'exp_T0I7ACMQLBMPG6K',
  mode: 'presenter',
  speechInputMode: 'auto'
});

experience.on('started', async () => {
  await experience.speech.speak({ text: 'What is your party size?' });
  const answer = await experience.listenOnce({ timeoutMs: 15_000 });
  console.log(answer.utteranceId, answer.text);
});

await experience.attach({ container: '#avatar' });

Only one listenOnce() may be active at a time. Timeout, abort, and permission failures reject without adding a final user message to conversation history. See Listen Once Capture.

conversationProcessor

Supply a browser function on Experience.startSession() to replace Liforma's managed LLM for that session. The player still orchestrates turns in conversation mode with speechInputMode: 'auto'; your function decides what the character should say next.

const experience = await Experience.startSession({
  experienceId: 'exp_T0I7ACMQLBMPG6K',
  mode: 'conversation',
  speechInputMode: 'auto',
  conversationProcessor: async ({ text, conversation, signal }) => {
    if (signal.aborted) throw new DOMException('Aborted', 'AbortError');
    if (text.toLowerCase().includes('checkout')) {
      return 'Checkout is at 11am. Need a late checkout?';
    }
    if (text.toLowerCase().includes('breakfast')) {
      return 'Breakfast is served until 10 in the lounge.';
    }
    return 'I can help with checkout, breakfast, or directions.';
  }
});

experience.on('conversationProcessorError', (evt) => {
  console.error('Processor failed', evt.data.utteranceId, evt.data.message);
});

await experience.attach({ container: '#avatar' });

The processor receives an immutable conversation snapshot, the finalized user text, utteranceId, and an AbortSignal cancelled when speak({ behavior: 'interrupt' }) or destroy() aborts in-flight work. Return a string, an object with text (and optional characterId, behavior), or an AsyncIterable<string> for streamed replies.

conversationProcessor: async function* ({ text, signal }) {
  const chunks = buildStoryChunks(text);
  for (const chunk of chunks) {
    if (signal.aborted) throw new DOMException('Aborted', 'AbortError');
    yield chunk;
  }
}

Processor failures emit conversationProcessorError and do not fall back to the managed LLM. Full walkthrough: Custom Conversation Processor.

setFit()

Change scene framing after attach() without reminting: full (default), medium (bust window), or face (inset / PIP). Safe as soon as the player is attached; does not wait for started.

await experience.attach({ container: document.querySelector('#avatar') });

// Later — e.g. when shrinking the host into a PIP tile
await experience.setFit('face');
await experience.setFit('full');

setInteraction() / getInteraction()

Patch live interaction settings after attach(). Three independent layers: state (playCharacterSpeech, showCharacterSpeech, micInputEnabled, textInputEnabled, showUserSpeech, visibleTurns), controls (whether the Voice, Captions, Microphone, and Text input buttons are shown), and capability (allowMicInput, allowTextInput, allowRewind). Hiding a control never changes state. showUserSpeech: false hides user bubbles and the listening graphic; there is no user toggle. Returns the player-resolved ExperienceInteraction. Subscribe to interactionChange for player-driven updates.

await experience.attach({ container: document.querySelector('#avatar') });

await experience.setInteraction({
  showCharacterSpeech: true,
  textInputEnabled: true,
  playCharacterSpeech: false,
  controls: { captions: false }
});

const interaction = await experience.getInteraction();

focusCharacter()

Presentation-only camera or tile emphasis on a multi-Character node. Pass the active-cast characterId. The call is a no-op when the active node has one Character. It does not choose who speaks — conversational speakers come from the engine-validated cast turn. Subscribe to characterFocusChanged for focus updates.

getConversation() and getLastTurn()

Read the flat in-session history as an ordered ConversationMessage[]. Each message has messageId, turnId, role (user or assistant), text, source (user, llm, speak, or opening), status, timestamp, and optional characterId for assistant lines.

const history = experience.getConversation();
const lastTurn = experience.getLastTurn();

experience.on('conversationUpdate', (evt) => {
  console.log('History length', evt.data.length);
});

getConversation() returns the current snapshot. getLastTurn() returns messages sharing the latest turnId. Subscribe to conversationUpdate for push updates when history changes.

For scripted lessons with manual Start/Stop, see Guided Scripted Practice and the guided-practice example on examples.liforma.ai.

Rendering is separate

Session creation and rendering configuration are independent. Attach the experience to a container to mount the player, display the avatar, and present its startup control.