liforma docs

ElevenLabs Agents → experience.speech

Pipe ElevenLabs agent audio into Liforma with @liforma/client/elevenlabs.

Idea in one sentence

Keep ElevenLabs Agents as the speech-to-speech brain, and use connectElevenLabsAgent from @liforma/client/elevenlabs to drive the Liforma avatar from agent PCM.

Copy into your product: the runnable example’s helloByo.ts / helloByo.js (startByoSpeech) is a thin wrapper over that helper — DemoApp / page UI is scaffolding only.

Install

npm install @liforma/client @elevenlabs/client

Checklist

  1. Mount a Liforma Experience with externalSpeechAudio (often mode="presenter", speechInputMode="off" when ElevenLabs owns the mic).
  2. Wait until the player has started (audio unlocked inside the iframe).
  3. Call connectElevenLabsAgent(experience, { signedUrl }) — the helper mutes ElevenLabs' speaker, locks sample rate from agent_output_audio_format (never guesses — missing format surfaces via onError), chunks PCM, and forwards transcript for lipsync.
  4. Mint the signed URL on your server (never ship ElevenLabs API keys to production browsers).
  5. Call bridge.end() when the conversation finishes.

Example

elevenlabs-liforma.ts
// After Experience is started (audio unlocked):
// Or copy helloByo.ts from examples/elevenlabs-embed → startByoSpeech(experience, { signedUrl })
import { connectElevenLabsAgent } from '@liforma/client/elevenlabs';

// Prefer a signed URL from your server in production:
// const signedUrl = await fetch('/api/elevenlabs-signed-url', {
//   method: 'POST',
//   headers: { 'Content-Type': 'application/json' },
//   body: JSON.stringify({ agentId: 'YOUR_AGENT_ID' })
// }).then(async (r) => {
//   const data = await r.json();
//   if (!r.ok) throw new Error(data.error ?? r.statusText);
//   return data.signedUrl as string;
// });

const bridge = await connectElevenLabsAgent(experience, {
  // signedUrl,
  agentId: 'YOUR_AGENT_ID' // demos / public agents
});

// … later
await bridge.end();

Production tip: pass a signed URL from your server instead of a bare agentId + API key in the browser.

Runnable example: ElevenLabs embed on examples.liforma.ai · live demo.

Using ElevenLabs WebRTC instead?

With WebRTC, ElevenLabs plays audio via LiveKit tracks and does not emit the same onAudio PCM stream. Bridge the remote track with the LiveKit guide.

Advanced: raw ConvAI WebSocket

Only if you are not using @elevenlabs/client / connectElevenLabsAgent. You must handle ping/pong and parse event types yourself.

elevenlabs-raw-ws.ts
// Advanced: raw ConvAI WebSocket (no @elevenlabs/client).
// Docs: https://elevenlabs.io/docs/eleven-agents/libraries/web-sockets
// Prefer connectElevenLabsAgent from @liforma/client/elevenlabs unless you need full control.

function base64ToArrayBuffer(b64: string): ArrayBuffer {
  const bin = atob(b64);
  const out = new Uint8Array(bin.length);
  for (let i = 0; i < bin.length; i++) out[i] = bin.charCodeAt(i);
  return out.buffer;
}

type Turn = {
  utterance: ReturnType<typeof experience.speech.createUtterance>;
  writes: Promise<void>;
};

let sampleRate = 16_000;
let turn: Turn | null = null;

const ws = new WebSocket(SIGNED_CONVAI_URL); // from your backend

ws.onmessage = (ev) => {
  const msg = JSON.parse(String(ev.data));

  if (msg.type === 'conversation_initiation_metadata') {
    const fmt = msg.conversation_initiation_metadata_event?.agent_output_audio_format;
    const m = /^pcm_(\d+)$/.exec(fmt ?? '');
    if (m) sampleRate = Number(m[1]);
    return;
  }

  if (msg.type === 'ping') {
    const eventId = msg.ping_event?.event_id;
    const delayMs = Number(msg.ping_event?.ping_ms ?? 0);
    const reply = () => ws.send(JSON.stringify({ type: 'pong', event_id: eventId }));
    if (delayMs > 0) setTimeout(reply, delayMs);
    else reply();
    return;
  }

  if (msg.type === 'interruption') {
    const current = turn;
    turn = null;
    void (current
      ? current.utterance.cancel()
      : experience.speech.interrupt({ scope: 'active' }));
    return;
  }

  if (msg.type === 'audio') {
    const b64 = msg.audio_event?.audio_base_64 as string | undefined;
    if (!b64) return;
    if (!turn) {
      const utterance = experience.speech.createUtterance({
        format: { encoding: 'pcm_s16le', sampleRate, channels: 1 },
        queue: 'replace-active'
      });
      turn = { utterance, writes: Promise.resolve() };
    }
    const u = turn.utterance;
    const chunk = base64ToArrayBuffer(b64);
    turn.writes = turn.writes.then(() => u.write(chunk)).catch(console.error);
    return;
  }

  if (msg.type === 'agent_response_complete') {
    // Enable this client event on the agent if you need it.
    const current = turn;
    turn = null;
    if (!current) return;
    void current.writes.then(() => current.utterance.close({ history: 'none' }));
  }
};

Not the migration package

@liforma/client/elevenlabs is bring-your-own-voice: keep ElevenLabs Agents and animate a Liforma avatar. To replace ElevenLabs Agents with a Liforma Experience, see Migrate from ElevenLabs (@liforma/elevenlabs-compatible).