Chapter 15 · Speech Transcription and Synthesis

Chapter goals:
Master the transcribe function for converting audio to text
Understand the experimental streaming transcription experimental_streamTranscribe and its适用 scenarios
Master generateSpeech for generating speech from text
Learn to handle download limits, abort signals, and speech-related error types

15.1 Audio Transcription

AI SDK provides the transcribe function, using a transcription model to transcribe audio.

import { transcribe, createGateway } from 'ai';
import { readFile } from 'fs/promises';
import 'dotenv/config';

const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY ?? '' });

const transcript = await transcribe({
  model: gateway.transcription('openai/whisper-1'),
  audio: await readFile('audio.mp3'),
});

The audio property can be a Uint8Array, ArrayBuffer, Buffer, string (base64-encoded audio data), or URL.

Access the generated transcript results:

const text = transcript.text; // transcript text e.g. "Hello, world!"
const segments = transcript.segments; // array of segments with start and end times, if available
const language = transcript.language; // language of the transcript e.g. "en", if available
const durationInSeconds = transcript.durationInSeconds; // duration of the transcript in seconds, if available

15.2 Streaming Transcription (Experimental)

Streaming transcription is an experimental feature.

When you hold real-time raw audio and need to receive transcription updates before the full audio stream ends, use experimental_streamTranscribe. This function uses a streaming-capable transcription model; provider-specific behavior is configured through providerOptions, and the streaming operation itself is selected by the function:

import { createGateway } from 'ai';
import { experimental_streamTranscribe as streamTranscribe } from 'ai';
import 'dotenv/config';

const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY ?? '' });

const result = streamTranscribe({
  model: gateway.transcription('openai/gpt-realtime-whisper'),
  audio: audioStream, // ReadableStream<Uint8Array | string>
  inputAudioFormat: { type: 'audio/pcm', rate: 24000 },
  providerOptions: {
    openai: {
      language: 'en',
      streaming: {
        delay: 'low',
      },
    },
  },
});

for await (const part of result.fullStream) {
  if (part.type === 'transcript-delta') {
    process.stdout.write(part.delta);
  }

  if (part.type === 'transcript-partial') {
    console.log('partial:', part.text);
  }

  if (part.type === 'transcript-final') {
    console.log('final:', part.text);
  }
}

console.log(await result.text);

fullStream is a single-consumer real-time stream that can only be accessed once. When you need both stream slices and the final result, access fullStream first, then await the result promise during or after consumption. If you access the result promise first, the stream is consumed internally and fullStream becomes unavailable — this avoids retaining an unbounded replay buffer for real-time audio.

Access final transcript metadata:

const text = await result.text; // final transcript text
const segments = await result.segments; // final segments with timing, if available
const language = await result.language; // language of the transcript, if available
const durationInSeconds = await result.durationInSeconds; // duration in seconds, if available

The audio stream must contain raw audio chunks: Uint8Array chunks are raw bytes, string chunks are base64-encoded raw bytes. Be sure to set inputAudioFormat to match the chunks you send.

The official docs note: string model IDs are resolved through the global default provider (AI Gateway by default). AI Gateway提供 streaming transcription for supported models (e.g., openai/gpt-realtime-whisper, elevenlabs/eleven-scribe-2-realtime, xai/grok-stt), so string IDs work. Streaming transcription models vary by provider: OpenAI uses gpt-realtime-whisper; Cartesia uses Ink 2 (cartesia.transcription('ink-2')); ElevenLabs uses Scribe v2 Realtime; xAI uses the same xai.transcription() model, and experimental_streamTranscribe selects its WebSocket STT transport.

import { xai } from '@ai-sdk/xai';
import { experimental_streamTranscribe as streamTranscribe } from 'ai';

const result = streamTranscribe({
  model: xai.transcription(),
  audio: audioStream,
  inputAudioFormat: { type: 'audio/pcm', rate: 16000 },
  providerOptions: {
    xai: {
      language: 'en',
      keyterm: ['AI SDK', 'Grok'],
      streaming: {
        interimResults: true,
        endpointing: 500,
      },
    },
  },
});

Some providers' direct streaming STT requires WebSocket headers — in such runtime environments, pass the provider-specific webSocket implementation when creating the provider.

15.3 Transcription Settings

Provider-Specific Settings

Transcription models typically have provider- or model-specific settings, configured via the providerOptions parameter:

import { transcribe, createGateway } from 'ai';
import { readFile } from 'fs/promises';
import 'dotenv/config';

const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY ?? '' });

const transcript = await transcribe({
  model: gateway.transcription('openai/whisper-1'),
  audio: await readFile('audio.mp3'),
  providerOptions: {
    openai: {
      timestampGranularities: ['word'],
    },
  },
});

Download Size Limits

When audio is a URL, the SDK downloads the file with a default 2 GiB cap. Customize with createDownload:

import { transcribe, createDownload, createGateway } from 'ai';
import 'dotenv/config';

const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY ?? '' });

const transcript = await transcribe({
  model: gateway.transcription('openai/whisper-1'),
  audio: new URL('https://example.com/audio.mp3'),
  download: createDownload({ maxBytes: 50 * 1024 * 1024 }), // 50 MB limit
});

You can also provide a fully custom download function:

const transcript = await transcribe({
  model: gateway.transcription('openai/whisper-1'),
  audio: new URL('https://example.com/audio.mp3'),
  download: async ({ url }) => {
    const res = await myAuthenticatedFetch(url);
    return {
      data: new Uint8Array(await res.arrayBuffer()),
      mediaType: res.headers.get('content-type') ?? undefined,
    };
  },
});

Downloading past the size limit throws a DownloadError:

import { transcribe, DownloadError, createGateway } from 'ai';
import 'dotenv/config';

const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY ?? '' });

try {
  await transcribe({
    model: gateway.transcription('openai/whisper-1'),
    audio: new URL('https://example.com/audio.mp3'),
  });
} catch (error) {
  if (DownloadError.isInstance(error)) {
    console.log('Download failed:', error.message);
  }
}

Abort Signals and Timeouts

transcribe accepts an optional AbortSignal typed abortSignal parameter, which can abort the transcription or set a timeout — combined with URL downloads it prevents long-hanging requests:

import { transcribe, createGateway } from 'ai';
import 'dotenv/config';

const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY ?? '' });

const transcript = await transcribe({
  model: gateway.transcription('openai/whisper-1'),
  audio: new URL('https://example.com/audio.mp3'),
  abortSignal: AbortSignal.timeout(5000), // Abort after 5 seconds
});

15.4 Speech Synthesis

AI SDK provides the generateSpeech function, using a speech model to generate audio from text.

import { generateSpeech, createGateway } from 'ai';
import 'dotenv/config';

const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY ?? '' });

const audio = await generateSpeech({
  model: gateway.speech('openai/tts-1'),
  text: 'Hello, world!',
  voice: 'alloy',
});

Language Settings

You can specify the language for speech generation (depending on provider support). The official example uses LMNT to demonstrate the language parameter:

import { generateSpeech } from 'ai';
import { lmnt } from '@ai-sdk/lmnt';

const audio = await generateSpeech({
  model: lmnt.speech('aurora'),
  text: 'Hola, mundo!',
  language: 'es', // Spanish
});

Access the generated audio data:

const audioData = result.audio.uint8Array; // audio data as Uint8Array
// or
const audioBase64 = result.audio.base64; // audio data as base64 string

15.5 Speech Synthesis Settings

Provider-Specific Settings

Use providerOptions to set model-specific options:

import { generateSpeech, createGateway } from 'ai';
import 'dotenv/config';

const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY ?? '' });

const audio = await generateSpeech({
  model: gateway.speech('openai/tts-1'),
  text: 'Hello, world!',
  providerOptions: {
    openai: {
      // ...
    },
  },
});

Abort Signals, Custom Headers, and Warnings

generateSpeech also accepts an optional abortSignal parameter to abort generation or set a timeout; an optional headers parameter to add custom headers to the request; unsupported parameters and warnings appear in the warnings property:

import { generateSpeech, createGateway } from 'ai';
import 'dotenv/config';

const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY ?? '' });

const audio = await generateSpeech({
  model: gateway.speech('openai/tts-1'),
  text: 'Hello, world!',
  abortSignal: AbortSignal.timeout(1000), // Abort after 1 second
});

import { generateSpeech, createGateway } from 'ai';
import 'dotenv/config';

const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY ?? '' });

const audio = await generateSpeech({
  model: gateway.speech('openai/tts-1'),
  text: 'Hello, world!',
});

const warnings = audio.warnings;

Error Handling

When generateSpeech cannot produce valid audio, it throws AI_NoSpeechGeneratedError. Possible causes: the model failed to generate a response, or it generated an unparseable response.

The error object retains the following information for debugging: responses (speech model response metadata, including timestamps, model, headers) and cause (root cause):

import { generateSpeech, NoSpeechGeneratedError, createGateway } from 'ai';
import 'dotenv/config';

const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY ?? '' });

try {
  await generateSpeech({
    model: gateway.speech('openai/tts-1'),
    text: 'Hello, world!',
  });
} catch (error) {
  if (NoSpeechGeneratedError.isInstance(error)) {
    console.log('AI_NoSpeechGeneratedError');
    console.log('Cause:', error.cause);
    console.log('Responses:', error.responses);
  }
}

15.6 Chapter Summary

transcribe is the unified audio transcription entry: audio supports Uint8Array / ArrayBuffer / Buffer / base64 string / URL; results include text, segments, language, durationInSeconds;
experimental_streamTranscribe is for real-time raw audio; fullStream is single-consumer — observe the "access fullStream first, then await result" ordering constraint;
URL audio downloads default to a 2 GiB cap; tighten with createDownload({ maxBytes }) or pass a fully custom download function; exceeding the limit throws DownloadError;
generateSpeech uses a speech model to turn text into audio, supporting voice, language, providerOptions, abortSignal, headers;
Speech generation failure throws NoSpeechGeneratedError; use isInstance to check and read cause / responses.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. Which of the following is NOT supported as an input for transcribe audio?

2. Regarding experimental_streamTranscribe's fullStream, which statement is correct?

3. What happens when audio is a URL and the file exceeds the download size limit?

4. What error type is thrown when generateSpeech cannot produce valid audio?

🛠️ Hands-On Exercises

Find a local mp3; transcribe it with transcribe + AI Gateway; print the full text, language code, and duration; iterate segments to output each segment's start/end times.
Use createDownload({ maxBytes }) to tighten the download cap to 5 MB; then发起 a transcription against a large remote audio file; catch the DownloadError and print the error message.
Use generateSpeech to synthesize a Chinese text passage into audio (try different voice values); write the uint8Array to a local file and play it back to compare effects.

Next: Chapter 16 · Chatbot: useChat and Stream Protocol

Chapter 15 · Speech Transcription and Synthesis ​

15.1 Audio Transcription ​

15.2 Streaming Transcription (Experimental) ​

15.3 Transcription Settings ​

Provider-Specific Settings ​

Download Size Limits ​

Abort Signals and Timeouts ​

15.4 Speech Synthesis ​

Language Settings ​

15.5 Speech Synthesis Settings ​

Provider-Specific Settings ​

Abort Signals, Custom Headers, and Warnings ​

Error Handling ​

15.6 Chapter Summary ​

🧪 随堂测验

🛠️ Hands-On Exercises ​

Chapter 15 · Speech Transcription and Synthesis

15.1 Audio Transcription

15.2 Streaming Transcription (Experimental)

15.3 Transcription Settings

Provider-Specific Settings

Download Size Limits

Abort Signals and Timeouts

15.4 Speech Synthesis

Language Settings

15.5 Speech Synthesis Settings

Provider-Specific Settings

Abort Signals, Custom Headers, and Warnings

Error Handling

15.6 Chapter Summary

🛠️ Hands-On Exercises