liforma docs

OpenAI → experience.speech

Pipe OpenAI Realtime (or classic TTS) into Liforma with @liforma/client/openai.

Idea in one sentence

Keep OpenAI Realtime as the speech-to-speech brain, and use connectOpenAiRealtime (WebSocket) or connectOpenAiRealtimeWebRtc (preferred browser media path) from @liforma/client/openai to drive the Liforma avatar (+ transcript for lipsync).

Copy into your product: the runnable example’s helloByo.ts / helloByo.js (startByoSpeech) is a thin wrapper over connectOpenAiRealtime — DemoApp / page UI is scaffolding only.

Install

npm install @liforma/client

Checklist

  1. Mount a Liforma Experience with externalSpeechAudio (often mode="presenter", speechInputMode="off" when OpenAI owns the mic).
  2. Wait until the player has started (audio unlocked inside the iframe).
  3. Mint an ephemeral Realtime client secret on your server (never ship OPENAI_API_KEY to production browsers).
  4. Call connectOpenAiRealtime(experience, { ephemeralKey }) — the helper opens the Realtime WebSocket, streams mic PCM (default captureMic: true; pass mediaStream to reuse a stream), chunks agent audio into createUtterance, and forwards transcript for force-align lipsync. Unexpected disconnect cancels any active utterance.
  5. Call bridge.end() when the conversation finishes.

Example

openai-liforma.ts
// After Experience is started (audio unlocked):
// Or copy helloByo.ts from examples/openai-realtime-embed → startByoSpeech(experience, { ephemeralKey })
import { connectOpenAiRealtime } from '@liforma/client/openai';

// Mint an ephemeral client secret on your server (never ship OPENAI_API_KEY):
// const ephemeralKey = await fetch('/api/openai-realtime-session', {
//   method: 'POST',
//   headers: { 'Content-Type': 'application/json' },
//   body: JSON.stringify({})
// }).then(async (r) => {
//   const data = await r.json();
//   if (!r.ok) throw new Error(data.error ?? r.statusText);
//   return String(data.value ?? data.ephemeralKey ?? '');
// });

const bridge = await connectOpenAiRealtime(experience, {
  ephemeralKey, // from your mint route
  // captureMic: true,      // default — set false if the host owns the mic
  // mediaStream,           // optional existing stream instead of getUserMedia
  // instructions: 'You are a helpful voice assistant…',
  // model: 'gpt-realtime-2.1',
  // voice: 'marin'
});

// … later
await bridge.end();

Runnable coffee-barista demo: OpenAI Realtime embed on examples.liforma.ai (local port 4007) · live demo.

Realtime WebRTC (preferred browser media)

connectOpenAiRealtimeWebRtc uses OpenAI’s WebRTC peer connection: remote audio track → createUtterance({ track }), transcript on the oai-events data channel for force-align. Same ephemeral client-secret mint as the WebSocket helper. Do not also attach the remote track to an <audio> element.

openai-realtime-webrtc.ts
// Preferred OpenAI browser media path: Realtime over WebRTC.
// Docs: https://platform.openai.com/docs/guides/realtime-webrtc
// Remote track → createUtterance({ track }); transcript via oai-events data channel.

import { connectOpenAiRealtimeWebRtc } from '@liforma/client/openai';

// Mint an ephemeral client secret on your server (same as the WebSocket helper).
const bridge = await connectOpenAiRealtimeWebRtc(experience, {
  ephemeralKey,
  // instructions: 'You are a helpful voice assistant…',
  // captureMic: true,
});

// Do not also attach the remote track to an <audio> element.
await bridge.end();
When to use the WebSocket helper

connectOpenAiRealtime maps each agent turn onto discrete createUtterance / write / close (same pattern as ElevenLabs). Prefer WebRTC for browser media latency; keep WebSocket when you already terminate Realtime over WS or need per-chunk PCM control.

Advanced: server-terminated WebSocket proxy

Only if you terminate OpenAI on a server and forward PCM to the browser. Prefer connectOpenAiRealtime for the browser WebSocket path.

openai-realtime-proxy.ts
// Advanced: OpenAI Realtime WebSocket on your server, PCM forwarded to the browser.
// Docs: https://platform.openai.com/docs/guides/realtime-conversations
// Prefer connectOpenAiRealtime from @liforma/client/openai unless you terminate OpenAI on a server.
// Server holds the OpenAI key; browser only talks to your proxy + Liforma.

// --- Browser ---
type Turn = {
  utterance: ReturnType<typeof experience.speech.createUtterance>;
  writes: Promise<void>;
};
let turn: Turn | null = null;

yourProxy.onPcmChunk((chunk: ArrayBuffer) => {
  if (!turn) {
    const utterance = experience.speech.createUtterance({
      format: { encoding: 'pcm_s16le', sampleRate: 24_000, channels: 1 },
      queue: 'replace-active'
    });
    turn = { utterance, writes: Promise.resolve() };
  }
  const u = turn.utterance;
  turn.writes = turn.writes.then(() => u.write(chunk)).catch(console.error);
});

yourProxy.onResponseDone(async () => {
  const current = turn;
  turn = null;
  if (!current) return;
  await current.writes;
  await current.utterance.close({ history: 'none' });
});

yourProxy.onSpeechStarted(() => {
  const current = turn;
  turn = null;
  void (current ? current.utterance.cancel() : experience.speech.interrupt({ scope: 'active' }));
});

// --- Server (Node) ---
// ws to wss://api.openai.com/v1/realtime?...
// on response.output_audio.delta → decode base64 → forward ArrayBuffer to browser
// on response.output_audio.done / response.done → signal browser to close
// on input_audio_buffer.speech_started → signal barge-in
Classic TTS (one-shot)

response_format: 'pcm' is raw pcm_s16le at 24 kHz (no WAV header). Split the OpenAI call (server) from speech.play (browser).

openai-tts.ts
// Classic OpenAI TTS — split server fetch from browser play.
// Docs: https://platform.openai.com/docs/guides/text-to-speech
// response_format: "pcm" → raw pcm_s16le @ 24 kHz (no WAV header).

// --- Server (Node / BFF) ---
import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const speech = await openai.audio.speech.create({
  model: 'gpt-4o-mini-tts',
  voice: 'alloy',
  input: 'Welcome to the lesson.',
  response_format: 'pcm'
});
return new Response(await speech.arrayBuffer(), {
  headers: { 'Content-Type': 'application/octet-stream' }
});

// --- Browser ---
const res = await fetch('/api/tts', {
  method: 'POST',
  body: JSON.stringify({ text: 'Welcome to the lesson.' })
});
const pcm = new Uint8Array(await res.arrayBuffer());
await experience.speech.play({
  audio: {
    data: pcm,
    format: { encoding: 'pcm_s16le', sampleRate: 24_000, channels: 1 }
  },
  queue: 'append'
});

Session capability

Mint with externalSpeechAudio.