liforma docs

Deepgram → experience.speech

Pipe Deepgram Voice Agent audio into Liforma with @liforma/client/deepgram.

Idea in one sentence

Keep Deepgram Voice Agent as the speech-to-speech brain, and use connectDeepgramAgent from @liforma/client/deepgram to drive the Liforma avatar from agent PCM (+ transcript when available).

Copy into your product: the runnable example’s helloByo.ts / helloByo.js (startByoSpeech) is a thin wrapper over that helper — DemoApp / page UI is scaffolding only. Keep the same-origin WebSocket proxy in your BFF.

Install

npm install @liforma/client

Checklist

  1. Mount a Liforma Experience with externalSpeechAudio (often mode="presenter", speechInputMode="off" when Deepgram owns the mic).
  2. Wait until the player has started (audio unlocked inside the iframe).
  3. Expose a same-origin WebSocket proxy to wss://agent.deepgram.com/v1/agent/converse (browsers cannot set Authorization on WebSocket).
  4. Call connectDeepgramAgent(experience, { proxyUrl }) — the helper runs Welcome → Settings → SettingsApplied, streams mic PCM, and forwards binary agent audio into createUtterance.
  5. Call bridge.end() when the conversation finishes.

Example

deepgram-liforma.ts
// After Experience is started (audio unlocked):
// Or copy helloByo.ts from examples/deepgram-embed → startByoSpeech(experience, { proxyUrl })
import { connectDeepgramAgent } from '@liforma/client/deepgram';

// Same-origin proxy to wss://agent.deepgram.com/v1/agent/converse (API key stays server-side):
// const proxyUrl = /* wss://your-origin/api/deepgram-agent */;

const bridge = await connectDeepgramAgent(experience, {
  proxyUrl,
  // agent: { listen / think / speak } — audio I/O formats are enforced by the helper
});

// … later
await bridge.end();

Runnable coffee-barista demo with a same-origin WebSocket proxy: Deepgram embed on examples.liforma.ai (local port 4008). The hosted page explains clone/run — Vercel serverless cannot host the long-lived WS proxy.

Required handshake (handled by the helper)
  1. Connect WebSocket
  2. Wait for Welcome — do not send anything before this
  3. Send Settings with audio.output.container: 'none' for headerless linear16
  4. Wait for SettingsApplied
  5. Then stream mic audio / accept agent binary PCM
Advanced: raw Voice Agent WebSocket

Only if you are not using connectDeepgramAgent. You must handle the handshake and binary frames yourself.

deepgram-agent-bridge.ts
// Advanced: own the Deepgram Voice Agent WebSocket yourself.
// Prefer connectDeepgramAgent from @liforma/client/deepgram unless you need full control.
// Docs: https://developers.deepgram.com/docs/voice-agent-message-flow
// Handshake: Welcome → Settings → SettingsApplied → then audio.
// Output must set container: "none" for headerless linear16 PCM.

const SAMPLE_RATE = 24_000;

type Turn = {
  utterance: ReturnType<typeof experience.speech.createUtterance>;
  writes: Promise<void>;
};

let turn: Turn | null = null;
let settingsApplied = false;

// Prefer a same-origin proxy (browser cannot set Authorization on WebSocket).
const ws = new WebSocket(YOUR_DEEPGRAM_AGENT_PROXY_URL);
ws.binaryType = 'arraybuffer';

const AGENT_SETTINGS = {
  type: 'Settings',
  audio: {
    input: { encoding: 'linear16', sample_rate: 16_000 },
    output: {
      encoding: 'linear16',
      sample_rate: SAMPLE_RATE,
      container: 'none'
    }
  },
  agent: {
    /* listen / think / speak — see Deepgram Settings docs */
  }
};

ws.onmessage = (ev) => {
  if (ev.data instanceof ArrayBuffer) {
    if (!settingsApplied) return; // ignore audio until SettingsApplied
    if (!turn) {
      const utterance = experience.speech.createUtterance({
        format: { encoding: 'pcm_s16le', sampleRate: SAMPLE_RATE, channels: 1 },
        queue: 'replace-active'
      });
      turn = { utterance, writes: Promise.resolve() };
    }
    const u = turn.utterance;
    const chunk = ev.data;
    turn.writes = turn.writes.then(() => u.write(chunk)).catch(console.error);
    return;
  }

  const msg = JSON.parse(String(ev.data));

  if (msg.type === 'Welcome') {
    // Do not send Settings until Welcome arrives.
    ws.send(JSON.stringify(AGENT_SETTINGS));
    return;
  }

  if (msg.type === 'SettingsApplied') {
    settingsApplied = true;
    return;
  }

  if (msg.type === 'UserStartedSpeaking') {
    const current = turn;
    turn = null;
    void (current
      ? current.utterance.cancel()
      : experience.speech.interrupt({ scope: 'active' }));
    return;
  }

  if (msg.type === 'AgentAudioDone') {
    const current = turn;
    turn = null;
    if (!current) return;
    void current.writes.then(() => current.utterance.close({ history: 'none' }));
  }
};
Aura / TTS-only

If you only use Deepgram TTS (not Voice Agent), fetch or stream the audio bytes and call speech.play — see other providers.

Session capability

Mint with externalSpeechAudio.