Experience API

Overview

Experience is the primary JavaScript API. Use it when you need programmatic control beyond the component — custom containers, event handling, presenter sequencing, or browser-owned conversation logic.

Start a session

main.js

import { Experience } from '@liforma/client';

const experience = await Experience.startSession({
  experienceId: 'exp_t0i7acmq'
});

await experience.attach({ container: '#avatar' });

Session inputs

Input	Use case
`{ experienceId }`	Browser embed — SDK calls `/v1/browser-sessions`
`{ launch }`	Advanced: client-fetched opaque launch (prefer `sessionEndpoint`)
`{ experienceId, sessionEndpoint }`	Server session — SDK POSTs to your same-origin route

Runtime context

runtimeContext is JSON you supply for this session: a name, account plan, current order, or other app facts. The opening line and system instructions can reference it with {{first_name}} or {{account.plan | default:"there"}}. Later turns receive the same JSON as untrusted reference data.

runtime-context.js

const session = await Experience.startSession({
experienceId: 'exp_…',
runtimeContext: {
  first_name: customer.firstName,
  account: { plan: customer.plan }
}
});

session.setRuntimeContext({ first_name: 'Charles', account: { plan: 'Pro' } });
session.updateRuntimeContext({ account: { plan: 'Team' } });

Experience instructions are the authored behaviour.
Knowledge is durable broader knowledge.
Runtime context is developer-supplied session data.
Website context is the current page when website assistant is on.
Conversation history is what the user and character have said.
Experience state is mutable game or workflow state.

Updating context affects the next managed turn. It does not remint the session, rewrite the spoken opening, or enter conversation history. It does not change tools, permissions, or capabilities. Authenticated variables remain a separate mint field and stay out of prompts.

Browser mint configuration

Pass these options to Experience.startSession(). They are forwarded when the SDK mints via the browser endpoint.

Option	Values	Purpose
`mode`	`conversation` \| `presenter`	Managed conversation or host-scripted presentation.
`speechInputMode`	`auto` \| `manual` \| `off`	Automatic turn detection, explicit listening boundaries, or no microphone.
`conversationProcessor`	Browser function	Replace the managed LLM with your own logic. The SDK derives custom response ownership internally and never serializes the function. See Custom Conversation Processor.
`onUserTranscript`	Callback	Observe partial and final STT updates. Observation only — returned values do not become speech automatically.
`speechOnly`	`boolean`	Run STT, TTS, and conversation without loading the avatar renderer or location scene assets. Same `experienceId` as a full embed. With `sessionEndpoint`, forward `speechOnly: true` in your BFF POST body. Default `false`.
`alreadyConverted`	`string[]` (e.g. `[‘registration’]`)	Conversion keys this play is already ineligible for. Snapshot at mint. Registration rate excludes these plays from the denominator. Do not fire `conversion(‘registration’)` for an already-registered visitor. Prefer setting this on server mint (`sessionEndpoint` / `POST /v1/sessions`).
`fit`	`full` \| `medium` \| `face`	Scene framing in the host container. Default `full` height-fits the whole avatar and location. `medium` is a bust window (0.75× face-mesh height above the oval, 2× below). `face` frames the mesh face oval with 0.5H above and below and at least 0.25W on each side (location background zooms too) — for inset / PIP layouts. Set on `attach()` or update live with `setFit()`. Not related to location video-call presentation.
`startButton`	Button label, accessibility, placement, variant, and appearance tokens	Customizes the player-owned startup control.
`theme`	`colors`, `typography`, `radiusPx`, `button`	Global player chrome (introduction, feedback, shared buttons, panels). Presentation only — does not remint. `startButton.appearance` still overrides the start CTA specifically.
`chrome`	`boolean`	Master switch for Liforma player controls. Omitted or `true` keeps the default chrome. `false` hides every control, including one an individual flag would show. Presentation only — does not remint.

Conversation sessions default to conversation, managed responses, and auto input. Presenter sessions default to presenter, manual responses, and manual input.

Supply conversationProcessor for browser-owned responses. Registered server processors (opaque processorId) are a separate upcoming capability.

Player-owned startup

The embedded player always renders and owns the actual startup click. Browsers require user activation inside the cross-origin player frame before it can unlock audio, so a button in the host page cannot replace this interaction.

Use the startButton property on attach() (or the matching <Experience> prop) to adapt the built-in control to your interface. Supported placement values are center, bottom-center, bottom-left, and bottom-right. Supported variants are primary, secondary, and minimal. Use theme for shared chrome (introduction, feedback, panels); startButton.appearance still wins on the start CTA.

Pass interaction on attach() (or the matching <Experience> / widget option) to merge defaults and control visibility before first paint. Nested controls patches are deep-merged. This does not remint.

chrome: false on attach() (or the matching <Experience> / widget option) hides every Liforma control around the experience: settings, theatre, full screen, close, voice, captions, microphone, text input, and history. An individual flag such as fullscreenButton: true does not bring that control back. Omitted or true leaves the per-button rules in place. Speech, listening, and the other SDK calls still work. The Start card stays on startButton, and end-of-session feedback stays on showFeedback. This does not remint.

await experience.attach({
container: document.querySelector('#avatar'),
chrome: false
});

To keep the Liforma chrome and hide one piece, leave chrome unset. Settings, theatre, full screen, and close use the same visibility shape as closeButton: true, false, or { visible: false }. settingsButton defaults on, theatreButton defaults off, and fullscreenButton uses the size heuristic when unset. Voice, captions, microphone, text input, and history stay on interaction.controls.

const experience = await Experience.startSession({
experienceId: 'exp_t0i7acmq',
mode: 'presenter'
});

experience.on('started', async (evt) => {
if (evt.data.mode === 'presenter') {
  await experience.speech.speak({ text: 'Welcome to the lesson.' });
}
});

await experience.attach({
container: document.querySelector('#avatar'),
theme: {
  colors: { primary: '#635bff', primaryText: '#ffffff' },
  radiusPx: 16
},
startButton: {
  label: 'Begin lesson',
  ariaLabel: 'Begin the guided lesson',
  placement: 'bottom-center', // center | bottom-center | bottom-left | bottom-right
  variant: 'primary', // primary | secondary | minimal
  appearance: {
    backgroundColor: '#635bff',
    textColor: '#ffffff',
    borderColor: '#8179ff',
    borderRadiusPx: 999,
    borderWidthPx: 2,
    size: 'large',
    fontFamily: 'system-ui, -apple-system, sans-serif',
    fontWeight: 700,
    // Or set an exact label size (10–48); overrides size font scaling:
    // fontSizePx: 22,
    shadow: 'soft'
  },
  icon: 'player-play', // Tabler id, or https://… / data:image/…
  iconPosition: 'left' // left | above
}
});

started fires after the player button is clicked and audio is unlocked. In presenter mode, it is the safe point for host code to call speak(). Calling speak(), startListening(), stopListening(), or listenOnce() before started throws a clear error. Button appearance is limited to the documented tokens; arbitrary CSS is not supported across the player frame. For a large pill button use size: ‘large’ with borderRadiusPx: 999, or set fontSizePx (10–48) for an exact label size. fontFamily accepts system / stack names available in the player iframe (host webfonts are not loaded there); fontWeight is 100–900. Optional icon accepts a Tabler Iconify id (e.g. player-play) or an absolute https://… / data:image/… URL; iconPosition is left (default) or above. Tabler icons inherit the button text colour. Site-relative paths are not supported (they would resolve against the player iframe origin).

Speech API

Call attach() first, wait for started, then use the methods below. See Events for characterSpeechStarted, userTranscript, userSpeechStarted, and related handlers.

Prerequisites

Speech methods throw if the session has not started. Register experience.on(‘started’, …) (or the onStart option on startSession) and only call speech methods after the player-owned start button unlocks audio.

`speech.speak()`

Queue animated character speech via Liforma TTS without invoking the managed LLM. Requires textSpeech. Resolves when playback completes.

const result = await experience.speech.speak({
  text: 'Repeat after me: Buenos días.',
  characterId: 'char_…', // optional — defaults to activeCharacterId
  queue: 'append' // optional — append (default), replace-active, or replace-all
});

console.log(result.utteranceId, result.durationMs, result.status);

Option	Type	Description
`text`	`string`	Line for the character to speak.
`characterId`	`string` (optional)	Defaults to `activeCharacterId` from the manifest.
`queue`	`append` \| `replace-active` \| `replace-all`	`append` waits for current speech (default). Replace policies interrupt before speaking.

`speech.play()` and `createUtterance()`

Bring-your-own audio. Requires externalSpeechAudio. Pass PCM, encoded bytes (audio/mpeg, …), or a CORS-open URL. Encoded/URL sources are decoded in the player, then lipsynced via speech-animation windows. See Bring your own voice.

When you have the spoken text, pass optional transcript on play, createUtterance, utterance.setTranscript(text), or close({ transcript }). Audio alone is enough; providing the text when available usually improves lipsync. Empty / whitespace is ignored.

// Encoded bytes — decoded in the player (not on api.liforma.ai)
await experience.speech.play({
  audio: { data: mp3Bytes, encoding: 'audio/mpeg' },
  queue: 'append'
});

// Or a CORS-open URL fetched + decoded inside the player iframe
await experience.speech.play({
  audio: { url: 'https://cdn.example.com/line.mp3' },
  queue: 'append'
});

stream-pcm.ts

type TurnState = {
  utterance: ReturnType<typeof experience.speech.createUtterance>;
  writes: Promise<void>;
};

const turns = new Map<string, TurnState>();

function beginTurn(turnId, transcript) {
  const utterance = experience.speech.createUtterance({
    format: { encoding: 'pcm_s16le', sampleRate: 24_000, channels: 1 },
    queue: 'replace-active',
    // Optional seed — pass spoken text when available (helps lipsync)
    ...(transcript ? { transcript } : {})
  });
  turns.set(turnId, { utterance, writes: Promise.resolve() });
}

function writeTurn(turnId, chunk) {
  const turn = turns.get(turnId);
  if (!turn) return;
  const u = turn.utterance;
  turn.writes = turn.writes
    .then(() => u.write(chunk))
    .catch((error) => {
      console.error('Unable to write speech audio', error);
      void u.cancel();
    });
}

async function endTurn(turnId, transcript) {
  const turn = turns.get(turnId);
  if (!turn) return;
  turns.delete(turnId);
  await turn.writes;
  await turn.utterance.close({ transcript, history: 'none' });
}

async function cancelTurn(turnId) {
  const turn = turns.get(turnId);
  turns.delete(turnId);
  if (turn) await turn.utterance.cancel();
}

`speech.interrupt()`

Returns settled SpeechResult[] for interrupted/cancelled utterances. Options: { scope: 'active' | 'all' }, { utteranceId }, or { characterId }.

`startListening()` and `stopListening()`

Open and close a logical microphone gate. The player acquires the physical microphone once during startup; stopListening() finalizes the utterance without stopping the media track. Use with a host microphoneActivation of tap_to_talk or push_to_talk so learner pauses do not end the utterance. An automatic experience rejects speechInputMode: ‘manual’ at mint.

await experience.startListening();
// Learner speaks; pauses do not end the utterance in manual mode.
const utterance = await experience.stopListening();
console.log(utterance.utteranceId, utterance.text);

stopListening() returns UtteranceResult with utteranceId and text. In managed conversation mode, a finalized manual utterance still triggers the configured processor. In presenter mode, the host decides what to do with the text.

`submitUserMessage({ text })`

Send the words from a host-owned text field as a user turn. This is available when chrome is false, when Liforma’s own text field is closed (textInputEnabled: false), and when the text control is hidden. It is rejected when the experience sets allowTextInput: false. The promise resolves when the player has queued the turn. Character speech and the reply still arrive through conversationUpdate. speech.speak() is a different action: it makes the character talk.

`toggleSettings()`

Open the player settings panel: microphone, speakers, listening mode, and language. toggleAudioSettings() remains on Experience as a deprecated alias. The panel-open event on the wire stays widgetAudioSettings. Prefer onWidgetSettings. If both onWidgetSettings and onWidgetAudioSettings are set, only onWidgetSettings runs.

`listenOnce()`

Convenience for automatic end-of-speech capture. Requires speechInputMode: ‘auto’. Opens one listening gate, waits for VAD to finalize a single utterance, then resolves with UtteranceResult. Use for quiz answers, voice forms, or presenter flows where the host owns the turn loop but does not want Start/Stop buttons.

const experience = await Experience.startSession({
  experienceId: 'exp_t0i7acmq',
  mode: 'presenter',
  speechInputMode: 'auto'
});

experience.on('started', async () => {
  await experience.speech.speak({ text: 'What is your party size?' });
  const answer = await experience.listenOnce({ timeoutMs: 15_000 });
  console.log(answer.utteranceId, answer.text);
});

await experience.attach({ container: '#avatar' });

Option	Type	Description
`timeoutMs`	`number` (optional)	Reject if no finalized utterance within this window (default 30s).
`signal`	`AbortSignal` (optional)	Abort the wait; rejects with `AbortError`.

Only one listenOnce() may be active at a time. Timeout, abort, and permission failures reject without adding a final user message to conversation history. See Listen Once Capture.

`conversationProcessor`

Supply a browser function on Experience.startSession() to replace Liforma’s managed LLM for that session. The player still orchestrates turns in conversation mode with speechInputMode: ‘auto’; your function decides what the character should say next.

const experience = await Experience.startSession({
  experienceId: 'exp_t0i7acmq',
  mode: 'conversation',
  speechInputMode: 'auto',
  conversationProcessor: async ({ text, conversation, signal }) => {
    if (signal.aborted) throw new DOMException('Aborted', 'AbortError');
    if (text.toLowerCase().includes('checkout')) {
      return 'Checkout is at 11am. Need a late checkout?';
    }
    if (text.toLowerCase().includes('breakfast')) {
      return 'Breakfast is served until 10 in the lounge.';
    }
    return 'I can help with checkout, breakfast, or directions.';
  }
});

experience.on('conversationProcessorError', (evt) => {
  console.error('Processor failed', evt.data.utteranceId, evt.data.message);
});

await experience.attach({ container: '#avatar' });

The processor receives an immutable conversation snapshot, the finalized user text, utteranceId, and an AbortSignal cancelled when speak({ behavior: ‘interrupt’ }) or destroy() aborts in-flight work. Return a string, an object with text (and optional characterId, behavior), or an AsyncIterable<string> for streamed replies.

conversationProcessor: async function* ({ text, signal }) {
  const chunks = buildStoryChunks(text);
  for (const chunk of chunks) {
    if (signal.aborted) throw new DOMException('Aborted', 'AbortError');
    yield chunk;
  }
}

Processor failures emit conversationProcessorError and do not fall back to the managed LLM. Full walkthrough: Custom Conversation Processor.

`setFit()`

Change scene framing after attach() without reminting: full (default), medium (bust window), or face (inset / PIP). Safe as soon as the player is attached; does not wait for started.

await experience.attach({ container: document.querySelector('#avatar') });

// Later — e.g. when shrinking the host into a PIP tile
await experience.setFit('face');
await experience.setFit('full');

`setInteraction()` / `getInteraction()`

Patch live interaction settings after attach(). Three independent layers: state (playCharacterSpeech, showCharacterSpeech, micInputEnabled, textInputEnabled, showUserSpeech, visibleTurns), controls (whether the Voice, Captions, Microphone, and Text input buttons are shown), and capability (allowMicInput, allowTextInput, allowRewind). Hiding a control never changes state. showUserSpeech: false hides user bubbles and the listening graphic; there is no user toggle. Returns the player-resolved ExperienceInteraction. Subscribe to interactionChange for player-driven updates.

await experience.attach({ container: document.querySelector('#avatar') });

await experience.setInteraction({
showCharacterSpeech: true,
textInputEnabled: true,
playCharacterSpeech: false,
controls: { captions: false }
});

const interaction = await experience.getInteraction();

`focusCharacter()`

Presentation-only camera or tile emphasis on a multi-Character node. Pass the active-cast characterId. The call is a no-op when the active node has one Character. It does not choose who speaks — conversational speakers come from the engine-validated cast turn. Subscribe to characterFocusChanged for focus updates.

`getConversation()` and `getLastTurn()`

Read the flat in-session history as an ordered ConversationMessage[]. Each message has messageId, turnId, role (user or assistant), text, source (user, llm, speak, or opening), status, timestamp, and optional characterId for assistant lines.

const history = experience.getConversation();
const lastTurn = experience.getLastTurn();

experience.on('conversationUpdate', (evt) => {
  console.log('History length', evt.data.length);
});

getConversation() returns the current snapshot. getLastTurn() returns messages sharing the latest turnId. Subscribe to conversationUpdate for push updates when history changes.

For scripted lessons with manual Start/Stop, see Guided Scripted Practice and the guided-practice example on examples.liforma.ai.

Rendering is separate

Session creation and rendering configuration are independent. Attach the experience to a container to mount the player, display the avatar, and present its startup control.

Experience API

Overview

Start a session

Session inputs

Runtime context

Browser mint configuration

Player-owned startup

Speech API

Prerequisites

speech.speak()

speech.play() and createUtterance()

speech.interrupt()

startListening() and stopListening()

submitUserMessage({ text })

toggleSettings()

listenOnce()

conversationProcessor

setFit()

setInteraction() / getInteraction()

focusCharacter()

getConversation() and getLastTurn()