Google → experience.speech

Idea in one sentence

Keep Gemini Live as the speech-to-speech brain, and use connectGeminiLive from @liforma/client/google to drive the Liforma avatar from agent PCM (+ outputTranscription for lipsync).

Copy into your product: the runnable example’s helloByo.ts / helloByo.js (startByoSpeech) is a thin wrapper over that helper — DemoApp / page UI is scaffolding only. Keep the same-origin WebSocket proxy in your BFF.

Runnable example

See examples.liforma.ai (examples/gemini-live-embed in the examples repo). The hosted page explains clone/run — Vercel serverless cannot host the long-lived WS proxy.

Install

npm install @liforma/client

Checklist

Mount a Liforma Experience with externalSpeechAudio (often mode="presenter", speechInputMode="off" when Gemini owns the mic).
Wait until the player has started (audio unlocked inside the iframe).
Expose a same-origin WebSocket proxy that terminates Gemini Live BidiGenerateContent (never ship Google API keys to browsers).
Call connectGeminiLive(experience, { proxyUrl }) — the helper streams mic PCM @ 16 kHz, writes model PCM @ 24 kHz into createUtterance, and closes on generationComplete / turnComplete.
Call bridge.end() when the conversation finishes.

Example

gemini-liforma.ts

// After Experience is started (audio unlocked):
// Or copy helloByo.ts from examples/gemini-live-embed → startByoSpeech(experience, { proxyUrl })
import { connectGeminiLive } from '@liforma/client/google';

// Proxy terminates Gemini Live BidiGenerateContent (never ship Google API keys):
// const proxyUrl = /* wss://your-origin/api/gemini-live */;

const bridge = await connectGeminiLive(experience, {
  proxyUrl
  // captureMic: true // default — streams PCM @ 16 kHz as realtimeInput.mediaChunks
});

// … later
await bridge.end();

Advanced: raw Gemini Live proxy messages

Only if you are not using connectGeminiLive. Accumulate outputTranscription independently of turnComplete ordering.

gemini-live-bridge.ts

// Advanced: own the Gemini Live proxy WebSocket yourself.
// Prefer connectGeminiLive from @liforma/client/google unless you need full control.
// Docs: https://ai.google.dev/gemini-api/docs/live-api
// outputTranscription is delivered independently — accumulate it; do not read it only on turnComplete.

function base64ToArrayBuffer(b64: string): ArrayBuffer {
  const bin = atob(b64);
  const out = new Uint8Array(bin.length);
  for (let i = 0; i < bin.length; i++) out[i] = bin.charCodeAt(i);
  return out.buffer;
}

type Turn = {
  utterance: ReturnType<typeof experience.speech.createUtterance>;
  writes: Promise<void>;
  transcript: string;
};

const SAMPLE_RATE = 24_000;
let turn: Turn | null = null;

function beginTurn(): Turn {
  const utterance = experience.speech.createUtterance({
    format: { encoding: 'pcm_s16le', sampleRate: SAMPLE_RATE, channels: 1 },
    queue: 'replace-active'
  });
  turn = { utterance, writes: Promise.resolve(), transcript: '' };
  return turn;
}

async function finishTurn(): Promise<void> {
  const current = turn;
  turn = null;
  if (!current) return;
  await current.writes;
  await current.utterance.close({
    transcript: current.transcript || undefined,
    history: 'none'
  });
}

ws.onmessage = (ev) => {
  const msg = JSON.parse(String(ev.data));
  const content = msg.serverContent;
  if (!content) return;

  if (content.interrupted) {
    const current = turn;
    turn = null;
    void (current
      ? current.utterance.cancel()
      : experience.speech.interrupt({ scope: 'active' }));
    return;
  }

  // Independent of modelTurn / turnComplete ordering
  if (content.outputTranscription?.text) {
    const current = turn ?? beginTurn();
    current.transcript += content.outputTranscription.text;
    void current.utterance.setTranscript(current.transcript);
  }

  for (const part of content.modelTurn?.parts ?? []) {
    const b64 = part.inlineData?.data as string | undefined;
    if (!b64) continue;
    const current = turn ?? beginTurn();
    const chunk = base64ToArrayBuffer(b64);
    const u = current.utterance;
    current.writes = current.writes.then(() => u.write(chunk)).catch(console.error);
  }

  // Prefer generationComplete when present (audio generation finished).
  // turnComplete can lag while the API assumes client-side playback timing.
  if (content.generationComplete || content.turnComplete) {
    void finishTurn();
  }
};

Cloud TTS (one-shot)

LINEAR16 responses include a WAV header. Pass them as encoded audio/wav — do not treat the bytes as raw pcm_s16le unless you strip the header yourself.

google-tts.ts

// Google Cloud Text-to-Speech — server synthesizes; browser plays.
// Docs: https://cloud.google.com/text-to-speech/docs/reference/rest/v1/text/synthesize
// Non-streaming LINEAR16 includes a WAV header → pass as audio/wav.

// --- Server ---
import textToSpeech from '@google-cloud/text-to-speech';

const client = new textToSpeech.TextToSpeechClient();
const [response] = await client.synthesizeSpeech({
  input: { text: 'Welcome to the lesson.' },
  voice: { languageCode: 'en-US', name: 'en-US-Chirp3-HD-Charon' },
  audioConfig: {
    audioEncoding: 'LINEAR16',
    sampleRateHertz: 24_000
  }
});
return new Response(response.audioContent as Uint8Array, {
  headers: { 'Content-Type': 'audio/wav' }
});

// --- Browser ---
const res = await fetch('/api/google-tts', {
  method: 'POST',
  body: JSON.stringify({ text: 'Welcome to the lesson.' })
});
const wavBytes = new Uint8Array(await res.arrayBuffer());
await experience.speech.play({
  audio: { data: wavBytes, encoding: 'audio/wav' },
  queue: 'append'
});

For streamingSynthesize, every audio_content response is headerless LINEAR16 at 24 kHz. Open a createUtterance with pcm_s16le @ 24 kHz, write each chunk, then close.

Session capability

Mint with externalSpeechAudio.