LiveKit → experience.speech

Idea in one sentence

Keep LiveKit (Agents, rooms, or any remote participant) as your voice source, and use connectLiveKitAgent from @liforma/client/livekit to drive the Liforma avatar from the remote MediaStreamTrack.

This is not Liforma’s future transport: livekit adapter. Here LiveKit is your external voice stack; the Experience session still mints normally (http adapter today) with externalSpeechAudio.

Copy into your product: the runnable example’s helloByo.ts / helloByo.js (startByoSpeech) is a thin wrapper over connectLiveKitAgent — DemoApp / page UI is scaffolding only.

Runnable example

See examples.liforma.ai (examples/livekit-embed in the examples repo) · live demo.

Install

npm install @liforma/client livekit-client

Checklist

1. Mount a Liforma Experience with externalSpeechAudio (often mode="presenter", speechInputMode="off").
2. Wait until the player has started (audio unlocked inside the iframe).
3. Mint a LiveKit participant token on your server.
4. Call connectLiveKitAgent(experience, { url, token }) — the helper joins the room, bridges agent audio via createUtterance({ track }), forwards lk.transcription text streams into setTranscript for force-align (disable with enableTranscript: false), and interrupts only while a bridge is outstanding.
5. Do not also play that track through a LiveKit <audio> element — you would hear the voice twice.
6. Call bridge.end() when the conversation finishes.

Example

// After Experience is started (audio unlocked):
// Or copy helloByo.ts from examples/livekit-embed → startByoSpeech(experience, { url, token })
import { connectLiveKitAgent } from '@liforma/client/livekit';

// Mint a LiveKit participant token on your server:
// const { url, token } = await fetch('/api/livekit-token', { method: 'POST' }).then((r) => r.json());

const bridge = await connectLiveKitAgent(experience, {
  url,   // wss://…
  token  // from your mint route
  // enableTranscript: true,           // default — lk.transcription → setTranscript
  // transcriptionTopic: 'lk.transcription',
  // shouldBridgeParticipant: (p) => p.identity.startsWith('agent')
});

// … later
await bridge.end();

// Advanced: own the LiveKit Room yourself.
// Prefer connectLiveKitAgent from @liforma/client/livekit unless you need full control.
import { Room, RoomEvent, Track } from 'livekit-client';

// Host owns the LiveKit room; Liforma owns avatar playback + lipsync.
// Do not call track.attach() / render an <audio> element for the agent — that doubles the voice.
const room = new Room();
let activeUtterance: ReturnType<typeof experience.speech.createUtterance> | null = null;

room.on(RoomEvent.TrackSubscribed, (track, _publication, participant) => {
  if (track.kind !== Track.Kind.Audio) return;
  // Optional: only bridge the agent participant
  // if (!participant.identity.startsWith('agent')) return;

  const mediaTrack = track.mediaStreamTrack;
  activeUtterance = experience.speech.createUtterance({
    track: mediaTrack,
    sampleRate: 48_000,
    queue: 'replace-active'
  });
  // Optional: room.registerTextStreamHandler('lk.transcription', …)
  // → activeUtterance.setTranscript(text) for force-align lipsync
});

room.on(RoomEvent.TrackUnsubscribed, async (track) => {
  if (track.kind !== Track.Kind.Audio) return;
  activeUtterance = null;
  await experience.speech.interrupt({ scope: 'active' });
});

await room.connect(LIVEKIT_URL, USER_TOKEN);

Session capability

Mint with externalSpeechAudio (and usually speechAnimation). Capture runs in the host SDK; the player only receives PCM.