liforma docs

Bring your own voice

Drive Liforma lipsync from ElevenLabs, OpenAI, Deepgram, LiveKit, or any PCM/file audio stack.

What this is for

Keep your own TTS or speech-to-speech provider. Stream or play audio into experience.speech; Liforma owns playback timing and avatar mouth animation.

Two speech paths

APICapabilityAudio source
speech.speaktextSpeechLiforma TTS
speech.play / createUtteranceexternalSpeechAudioYour PCM, encoded bytes, or CORS-open URL

Unsupported operations throw UnsupportedSpeechOperationError immediately. Mint the session with externalSpeechAudio (and usually speechAnimation) when the host owns the voice.

One-shot PCM

play-pcm.js
await experience.speech.play({
  audio: {
    data: pcmS16leBytes,
    format: { encoding: 'pcm_s16le', sampleRate: 24_000, channels: 1 }
  },
  // Optional but recommended when you have the spoken text — usually improves lipsync.
  transcript: agentReplyText,
  queue: 'append' // append | replace-active | replace-all
});

Encoded file or CDN URL

Decode happens in the player (Web Audio). Api/models never fetch or transcode arbitrary media URLs. URL playback requires CORS that allows player.liforma.ai.

play-encoded.js
// Encoded bytes — decoded in the player (not on api.liforma.ai)
await experience.speech.play({
  audio: { data: mp3Bytes, encoding: 'audio/mpeg' },
  queue: 'append'
});

// Or a CORS-open URL fetched + decoded inside the player iframe
await experience.speech.play({
  audio: { url: 'https://cdn.example.com/line.mp3' },
  queue: 'append'
});

Live streaming

Use createUtterance when your vendor emits many PCM chunks per agent turn. Each provider guide maps real vendor events onto write / close / interrupt — start with ElevenLabs, OpenAI, Google, Deepgram, or LiveKit.

Lipsync quality floors

PathAudioTranscriptMouth quality
PCM helpers (createUtterance)Correct sample rate locked before writeYes (force-align)Highest — prefer this when the vendor exposes text
PCM without transcriptCorrect sample rateNoGood — free CTC / energy; less precise phone timing
Track helpers (createUtterance({ track }))Sample-rate hint (e.g. LiveKit 48 kHz)Yes when available (e.g. LiveKit lk.transcription)High with transcript; baseline free CTC without — prefer connectLiveKitAgent over bare speech.play

Lipsync behaviour

  • Audio starts after a short jitter buffer (~80–250 ms adaptive from chunk arrival variance; see BYO_PLAYBACK_GUIDANCE on @liforma/client/byo) — it never waits on cloud speech-animation.
  • Mouth starts with an energy fallback, then crossfades to tracked phone/JALI results.
  • Tracked animation uses limited-context HTTP windows to POST /v1/speech-animation/windows (player-internal).
  • When the spoken text is available, pass it as transcript on speech.play, createUtterance, setTranscript, and/or close. Audio alone works; including the text usually improves mouth timing and shapes.
  • Custom bridges should chunk PCM around 20–200 ms ( BYO_PLAYBACK_GUIDANCE.RECOMMENDED_CHUNK_MS_*) and never exceed MAX_PCM_CHUNK_BYTES (64 KiB) per write.

Mic ownership

PCM helpers that capture audio (openai, deepgram, google) share the same options: captureMic (default true) and optional mediaStream. Keep Liforma speechInputMode="off" so only one stack owns the mic. On unexpected disconnect, helpers cancel the active utterance so the player does not keep an orphaned turn.

Provider guides