# SkillPatch skill: azure-ai-voicelive-ts

This skill provides guidance for building real-time voice AI applications using the Azure AI Voice Live SDK for JavaScript/TypeScript. It covers authentication (Entra ID and API key), client setup, session configuration, and bidirectional WebSocket audio streaming. It targets Node.js and browser environments for use cases like voice assistants, conversational AI, and speech-to-speech applications.

You (the agent) were given this URL and asked to use this skill. This is a **public** skill — no SkillPatch account, API key, or prior setup is required. Two ways to use it:

**1. Use it right now** — the skill's complete file tree (SKILL.md + all reference files) is inlined below; read `SKILL.md` first, then follow it, consulting the other files as it directs.

**2. Install the exact package onto disk** (recommended if you can run a shell — this reproduces the skill byte-for-byte, including any binary assets that can't be inlined):

```bash
mkdir -p .claude/skills/azure-ai-voicelive-ts
curl -sSL https://skillpatch.dev/install_skill/azure-ai-voicelive-ts | tar -xz -C .claude/skills/
```

(`.claude/skills/` is Claude Code's convention; use whatever directory your agent loads skills from.)


---

## Skill files (3)

- `SKILL.md`
- `references/audio-streaming.md`
- `references/function-calling.md`


### `SKILL.md`

````markdown
---
name: azure-ai-voicelive-ts
description: |
  Azure AI Voice Live SDK for JavaScript/TypeScript. Build real-time voice AI applications with bidirectional WebSocket communication. Use for voice assistants, conversational AI, real-time speech-to-speech, and voice-enabled chatbots in Node.js or browser environments. Triggers: "voice live", "real-time voice", "VoiceLiveClient", "VoiceLiveSession", "voice assistant TypeScript", "bidirectional audio", "speech-to-speech JavaScript".
license: MIT
metadata:
  author: Microsoft
  version: "1.0.0"
  package: '@azure/ai-voicelive'
---

# @azure/ai-voicelive (JavaScript/TypeScript)

Real-time voice AI SDK for building bidirectional voice assistants with Azure AI in Node.js and browser environments.

## Installation

```bash
npm install @azure/ai-voicelive @azure/identity
# TypeScript users
npm install @types/node
```

**Current Version**: 1.0.0-beta.3

**Supported Environments**:
- Node.js LTS versions (20+)
- Modern browsers (Chrome, Firefox, Safari, Edge)

## Environment Variables

```bash
AZURE_VOICELIVE_ENDPOINT=https://<resource>.cognitiveservices.azure.com
# Optional: API key if not using Entra ID
AZURE_VOICELIVE_API_KEY=<your-api-key>
# Optional: Logging
AZURE_LOG_LEVEL=info
AZURE_TOKEN_CREDENTIALS=prod # Required only if DefaultAzureCredential is used in production
```

## Authentication

### Microsoft Entra Token Credential (Recommended)

```typescript
import { DefaultAzureCredential, ManagedIdentityCredential } from "@azure/identity";
import { VoiceLiveClient } from "@azure/ai-voicelive";

// Local dev: DefaultAzureCredential. Production: set AZURE_TOKEN_CREDENTIALS=prod or AZURE_TOKEN_CREDENTIALS=<specific_credential>
const credential = new DefaultAzureCredential({requiredEnvVars: ["AZURE_TOKEN_CREDENTIALS"]});
// Or use a specific credential directly in production:
// See https://learn.microsoft.com/javascript/api/overview/azure/identity-readme?view=azure-node-latest#credential-classes
// const credential = new ManagedIdentityCredential();
const endpoint = "https://your-resource.cognitiveservices.azure.com";

const client = new VoiceLiveClient(endpoint, credential);
```

### API Key

```typescript
import { AzureKeyCredential } from "@azure/core-auth";
import { VoiceLiveClient } from "@azure/ai-voicelive";

const endpoint = "https://your-resource.cognitiveservices.azure.com";
const credential = new AzureKeyCredential("your-api-key");

const client = new VoiceLiveClient(endpoint, credential);
```

## Client Hierarchy

```
VoiceLiveClient
└── VoiceLiveSession (WebSocket connection)
    ├── updateSession()      → Configure session options
    ├── subscribe()          → Event handlers (Azure SDK pattern)
    ├── sendAudio()          → Stream audio input
    ├── addConversationItem() → Add messages/function outputs
    └── sendEvent()          → Send raw protocol events
```

## Quick Start

```typescript
import { DefaultAzureCredential } from "@azure/identity";
import { VoiceLiveClient } from "@azure/ai-voicelive";

const credential = new DefaultAzureCredential({requiredEnvVars: ["AZURE_TOKEN_CREDENTIALS"]});
const endpoint = process.env.AZURE_VOICELIVE_ENDPOINT!;

// Create client and start session
const client = new VoiceLiveClient(endpoint, credential);
const session = await client.startSession("gpt-4o-mini-realtime-preview");

// Configure session
await session.updateSession({
  modalities: ["text", "audio"],
  instructions: "You are a helpful AI assistant. Respond naturally.",
  voice: {
    type: "azure-standard",
    name: "en-US-AvaNeural",
  },
  turnDetection: {
    type: "server_vad",
    threshold: 0.5,
    prefixPaddingMs: 300,
    silenceDurationMs: 500,
  },
  inputAudioFormat: "pcm16",
  outputAudioFormat: "pcm16",
});

// Subscribe to events
const subscription = session.subscribe({
  onResponseAudioDelta: async (event, context) => {
    // Handle streaming audio output
    const audioData = event.delta;
    playAudioChunk(audioData);
  },
  onResponseTextDelta: async (event, context) => {
    // Handle streaming text
    process.stdout.write(event.delta);
  },
  onInputAudioTranscriptionCompleted: async (event, context) => {
    console.log("User said:", event.transcript);
  },
});

// Send audio from microphone
function sendAudioChunk(audioBuffer: ArrayBuffer) {
  session.sendAudio(audioBuffer);
}
```

## Session Configuration

```typescript
await session.updateSession({
  // Modalities
  modalities: ["audio", "text"],
  
  // System instructions
  instructions: "You are a customer service representative.",
  
  // Voice selection
  voice: {
    type: "azure-standard",  // or "azure-custom", "openai"
    name: "en-US-AvaNeural",
  },
  
  // Turn detection (VAD)
  turnDetection: {
    type: "server_vad",      // or "azure_semantic_vad"
    threshold: 0.5,
    prefixPaddingMs: 300,
    silenceDurationMs: 500,
  },
  
  // Audio formats
  inputAudioFormat: "pcm16",
  outputAudioFormat: "pcm16",
  
  // Tools (function calling)
  tools: [
    {
      type: "function",
      name: "get_weather",
      description: "Get current weather",
      parameters: {
        type: "object",
        properties: {
          location: { type: "string" }
        },
        required: ["location"]
      }
    }
  ],
  toolChoice: "auto",
});
```

## Event Handling (Azure SDK Pattern)

The SDK uses a subscription-based event handling pattern:

```typescript
const subscription = session.subscribe({
  // Connection lifecycle
  onConnected: async (args, context) => {
    console.log("Connected:", args.connectionId);
  },
  onDisconnected: async (args, context) => {
    console.log("Disconnected:", args.code, args.reason);
  },
  onError: async (args, context) => {
    console.error("Error:", args.error.message);
  },
  
  // Session events
  onSessionCreated: async (event, context) => {
    console.log("Session created:", context.sessionId);
  },
  onSessionUpdated: async (event, context) => {
    console.log("Session updated");
  },
  
  // Audio input events (VAD)
  onInputAudioBufferSpeechStarted: async (event, context) => {
    console.log("Speech started at:", event.audioStartMs);
  },
  onInputAudioBufferSpeechStopped: async (event, context) => {
    console.log("Speech stopped at:", event.audioEndMs);
  },
  
  // Transcription events
  onConversationItemInputAudioTranscriptionCompleted: async (event, context) => {
    console.log("User said:", event.transcript);
  },
  onConversationItemInputAudioTranscriptionDelta: async (event, context) => {
    process.stdout.write(event.delta);
  },
  
  // Response events
  onResponseCreated: async (event, context) => {
    console.log("Response started");
  },
  onResponseDone: async (event, context) => {
    console.log("Response complete");
  },
  
  // Streaming text
  onResponseTextDelta: async (event, context) => {
    process.stdout.write(event.delta);
  },
  onResponseTextDone: async (event, context) => {
    console.log("\n--- Text complete ---");
  },
  
  // Streaming audio
  onResponseAudioDelta: async (event, context) => {
    const audioData = event.delta;
    playAudioChunk(audioData);
  },
  onResponseAudioDone: async (event, context) => {
    console.log("Audio complete");
  },
  
  // Audio transcript (what assistant said)
  onResponseAudioTranscriptDelta: async (event, context) => {
    process.stdout.write(event.delta);
  },
  
  // Function calling
  onResponseFunctionCallArgumentsDone: async (event, context) => {
    if (event.name === "get_weather") {
      const args = JSON.parse(event.arguments);
      const result = await getWeather(args.location);
      
      await session.addConversationItem({
        type: "function_call_output",
        callId: event.callId,
        output: JSON.stringify(result),
      });
      
      await session.sendEvent({ type: "response.create" });
    }
  },
  
  // Catch-all for debugging
  onServerEvent: async (event, context) => {
    console.log("Event:", event.type);
  },
});

// Clean up when done
await subscription.close();
```

## Function Calling

```typescript
// Define tools in session config
await session.updateSession({
  modalities: ["audio", "text"],
  instructions: "Help users with weather information.",
  tools: [
    {
      type: "function",
      name: "get_weather",
      description: "Get current weather for a location",
      parameters: {
        type: "object",
        properties: {
          location: {
            type: "string",
            description: "City and state or country",
          },
        },
        required: ["location"],
      },
    },
  ],
  toolChoice: "auto",
});

// Handle function calls
const subscription = session.subscribe({
  onResponseFunctionCallArgumentsDone: async (event, context) => {
    if (event.name === "get_weather") {
      const args = JSON.parse(event.arguments);
      const weatherData = await fetchWeather(args.location);
      
      // Send function result
      await session.addConversationItem({
        type: "function_call_output",
        callId: event.callId,
        output: JSON.stringify(weatherData),
      });
      
      // Trigger response generation
      await session.sendEvent({ type: "response.create" });
    }
  },
});
```

## Voice Options

| Voice Type | Config | Example |
|------------|--------|---------|
| Azure Standard | `{ type: "azure-standard", name: "..." }` | `"en-US-AvaNeural"` |
| Azure Custom | `{ type: "azure-custom", name: "...", endpointId: "..." }` | Custom voice endpoint |
| Azure Personal | `{ type: "azure-personal", speakerProfileId: "..." }` | Personal voice clone |
| OpenAI | `{ type: "openai", name: "..." }` | `"alloy"`, `"echo"`, `"shimmer"` |

## Supported Models

| Model | Description | Use Case |
|-------|-------------|----------|
| `gpt-4o-realtime-preview` | GPT-4o with real-time audio | High-quality conversational AI |
| `gpt-4o-mini-realtime-preview` | Lightweight GPT-4o | Fast, efficient interactions |
| `phi4-mm-realtime` | Phi multimodal | Cost-effective applications |

## Turn Detection Options

```typescript
// Server VAD (default)
turnDetection: {
  type: "server_vad",
  threshold: 0.5,
  prefixPaddingMs: 300,
  silenceDurationMs: 500,
}

// Azure Semantic VAD (smarter detection)
turnDetection: {
  type: "azure_semantic_vad",
}

// Azure Semantic VAD (English optimized)
turnDetection: {
  type: "azure_semantic_vad_en",
}

// Azure Semantic VAD (Multilingual)
turnDetection: {
  type: "azure_semantic_vad_multilingual",
}
```

## Audio Formats

| Format | Sample Rate | Use Case |
|--------|-------------|----------|
| `pcm16` | 24kHz | Default, high quality |
| `pcm16-8000hz` | 8kHz | Telephony |
| `pcm16-16000hz` | 16kHz | Voice assistants |
| `g711_ulaw` | 8kHz | Telephony (US) |
| `g711_alaw` | 8kHz | Telephony (EU) |

## Key Types Reference

| Type | Purpose |
|------|---------|
| `VoiceLiveClient` | Main client for creating sessions |
| `VoiceLiveSession` | Active WebSocket session |
| `VoiceLiveSessionHandlers` | Event handler interface |
| `VoiceLiveSubscription` | Active event subscription |
| `ConnectionContext` | Context for connection events |
| `SessionContext` | Context for session events |
| `ServerEventUnion` | Union of all server events |

## Error Handling

```typescript
import {
  VoiceLiveError,
  VoiceLiveConnectionError,
  VoiceLiveAuthenticationError,
  VoiceLiveProtocolError,
} from "@azure/ai-voicelive";

const subscription = session.subscribe({
  onError: async (args, context) => {
    const { error } = args;
    
    if (error instanceof VoiceLiveConnectionError) {
      console.error("Connection error:", error.message);
    } else if (error instanceof VoiceLiveAuthenticationError) {
      console.error("Auth error:", error.message);
    } else if (error instanceof VoiceLiveProtocolError) {
      console.error("Protocol error:", error.message);
    }
  },
  
  onServerError: async (event, context) => {
    console.error("Server error:", event.error?.message);
  },
});
```

## Logging

```typescript
import { setLogLevel } from "@azure/logger";

// Enable verbose logging
setLogLevel("info");

// Or via environment variable
// AZURE_LOG_LEVEL=info
```

## Browser Usage

```typescript
// Browser requires bundler (Vite, webpack, etc.)
import { VoiceLiveClient } from "@azure/ai-voicelive";
import { InteractiveBrowserCredential } from "@azure/identity";

// Use browser-compatible credential
const credential = new InteractiveBrowserCredential({
  clientId: "your-client-id",
  tenantId: "your-tenant-id",
});

const client = new VoiceLiveClient(endpoint, credential);

// Request microphone access
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const audioContext = new AudioContext({ sampleRate: 24000 });

// Process audio and send to session
// ... (see samples for full implementation)
```

## Best Practices

1. **Use `DefaultAzureCredential` for local development; use `ManagedIdentityCredential` or `WorkloadIdentityCredential` for production** — Never hardcode API keys
2. **Set both modalities** — Include `["text", "audio"]` for voice assistants
3. **Use Azure Semantic VAD** — Better turn detection than basic server VAD
4. **Handle all error types** — Connection, auth, and protocol errors
5. **Clean up subscriptions** — Call `subscription.close()` when done
6. **Use appropriate audio format** — PCM16 at 24kHz for best quality

## Reference Links

| Resource | URL |
|----------|-----|
| npm Package | https://www.npmjs.com/package/@azure/ai-voicelive |
| GitHub Source | https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/ai/ai-voicelive |
| Samples | https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/ai/ai-voicelive/samples |
| API Reference | https://learn.microsoft.com/javascript/api/@azure/ai-voicelive |

````


### `references/audio-streaming.md`

````markdown
# Audio Streaming Reference

Real-time audio capture and playback patterns for Azure AI Voice Live using the @azure/ai-voicelive TypeScript SDK.

## Overview

Voice Live requires bidirectional audio streaming over WebSocket. This reference covers browser microphone capture, PCM format conversion, and audio playback patterns for voice assistants.

## Audio Format Requirements

| Property | Default Value | Options |
|----------|---------------|---------|
| `inputAudioFormat` | `"pcm16"` | `"pcm16"`, `"g711_alaw"`, `"g711_ulaw"` |
| `outputAudioFormat` | `"pcm16"` | `"pcm16"` |
| Sample Rate (pcm16) | 24000 Hz | 8000, 16000, 24000 |
| Channels | Mono (1) | - |

### Sample Rate Inference

```typescript
function inferPcmSampleRateFromOutputFormat(outputAudioFormat: string | undefined): number {
  if (!outputAudioFormat) return 24000;
  const fmt = outputAudioFormat.toLowerCase();
  if (fmt === "pcm16") return 24000;
  if (fmt === "pcm16-16000hz") return 16000;
  if (fmt === "pcm16-8000hz") return 8000;
  return 24000;
}
```

## Browser Microphone Capture

### Pattern A: ScriptProcessorNode (Simple)

Simpler implementation but uses deprecated Web Audio API.

```typescript
export class SimpleAudioCapture {
  private audioContext?: AudioContext;
  private mediaStream?: MediaStream;
  private scriptProcessor?: ScriptProcessorNode;
  private dataCallback: (data: ArrayBuffer) => void;
  
  private readonly targetSampleRate = 24000;

  constructor(onData: (data: ArrayBuffer) => void) {
    this.dataCallback = onData;
  }

  async initialize(): Promise<void> {
    // Request microphone access
    this.mediaStream = await navigator.mediaDevices.getUserMedia({
      audio: {
        channelCount: 1,
        sampleRate: this.targetSampleRate,
        echoCancellation: true,
        noiseSuppression: true,
      },
    });

    // Create audio context at target sample rate
    this.audioContext = new AudioContext({ sampleRate: this.targetSampleRate });
    
    // Create processing chain
    const source = this.audioContext.createMediaStreamSource(this.mediaStream);
    this.scriptProcessor = this.audioContext.createScriptProcessor(4096, 1, 1);
    
    // Connect nodes
    source.connect(this.scriptProcessor);
    this.scriptProcessor.connect(this.audioContext.destination);
    
    // Process audio data
    this.scriptProcessor.onaudioprocess = (event) => {
      const inputData = event.inputBuffer.getChannelData(0);
      const pcm16Data = this.convertToPCM16(inputData);
      this.dataCallback(pcm16Data.buffer);
    };
  }

  private convertToPCM16(floatData: Float32Array): Int16Array {
    const pcm16 = new Int16Array(floatData.length);
    for (let i = 0; i < floatData.length; i++) {
      const sample = Math.max(-1, Math.min(1, floatData[i]));
      pcm16[i] = sample < 0 ? sample * 0x8000 : sample * 0x7fff;
    }
    return pcm16;
  }

  stop(): void {
    this.scriptProcessor?.disconnect();
    this.mediaStream?.getTracks().forEach((track) => track.stop());
    this.audioContext?.close();
  }
}
```

### Pattern B: AudioWorklet (Modern, Recommended)

Uses modern AudioWorklet API with better performance.

```typescript
// Main thread code
export class AudioWorkletCapture {
  private audioContext?: AudioContext;
  private mediaStream?: MediaStream;
  private workletNode?: AudioWorkletNode;
  private onAudioData: (data: Uint8Array) => void;

  constructor(onData: (data: Uint8Array) => void) {
    this.onAudioData = onData;
  }

  async start(): Promise<void> {
    const ac = new AudioContext();
    this.audioContext = ac;

    // Load AudioWorklet module
    const workletUrl = new URL("worklets/pcm16-downsampler.js", document.baseURI).toString();
    await ac.audioWorklet.addModule(workletUrl);

    // Get microphone stream
    // Disable browser processing - server handles VAD
    const stream = await navigator.mediaDevices.getUserMedia({
      audio: {
        echoCancellation: false,
        noiseSuppression: false,
        autoGainControl: false,
      },
    });
    this.mediaStream = stream;

    // Create processing chain
    const source = ac.createMediaStreamSource(stream);
    const worklet = new AudioWorkletNode(ac, "pcm16-downsampler");
    this.workletNode = worklet;

    // Receive processed PCM16 data from worklet
    worklet.port.onmessage = (e) => {
      const buffer = e.data as ArrayBuffer;
      const pcm16Bytes = new Uint8Array(buffer);
      this.onAudioData(pcm16Bytes);
    };

    // Connect with muted output to prevent feedback
    const gain = ac.createGain();
    gain.gain.value = 0;
    source.connect(worklet);
    worklet.connect(gain);
    gain.connect(ac.destination);
  }

  stop(): void {
    this.workletNode?.disconnect();
    this.mediaStream?.getTracks().forEach((track) => track.stop());
    this.audioContext?.close();
  }
}
```

### AudioWorklet Processor (pcm16-downsampler.js)

Save this file in `/public/worklets/pcm16-downsampler.js`:

```javascript
class Pcm16DownsamplerProcessor extends AudioWorkletProcessor {
  constructor() {
    super();
    this._targetSampleRate = 24000;
    this._inputSampleRate = sampleRate;  // Browser's native rate (e.g., 48000)
    this._ratio = this._inputSampleRate / this._targetSampleRate;
    this._carry = 0;
  }

  process(inputs) {
    const input = inputs[0];
    if (!input || input.length === 0) return true;

    const channel0 = input[0];
    if (!channel0 || channel0.length === 0) return true;

    // Downsample by averaging
    const outLength = Math.max(0, Math.floor((channel0.length - this._carry) / this._ratio));
    if (outLength === 0) return true;

    const pcm16 = new Int16Array(outLength);
    let outIndex = 0;
    let i = this._carry;

    while (outIndex < outLength) {
      const start = Math.floor(i);
      const end = Math.floor(i + this._ratio);

      let sum = 0, count = 0;
      for (let j = start; j < end && j < channel0.length; j++) {
        sum += channel0[j];
        count++;
      }

      const sample = count > 0 ? sum / count : 0;
      const clamped = Math.max(-1, Math.min(1, sample));
      pcm16[outIndex] = clamped < 0 ? clamped * 0x8000 : clamped * 0x7fff;

      outIndex++;
      i += this._ratio;
    }

    this._carry = i - channel0.length;
    if (this._carry < 0) this._carry = 0;

    // Transfer buffer to main thread
    this.port.postMessage(pcm16.buffer, [pcm16.buffer]);
    return true;
  }
}

registerProcessor("pcm16-downsampler", Pcm16DownsamplerProcessor);
```

## Audio Playback

### Pattern A: AudioBufferSourceNode Queue

Simple queue-based playback using AudioBufferSourceNode.

```typescript
export class Pcm16Player {
  private readonly audioContext: AudioContext;
  private nextStartTime = 0;
  private sources: AudioBufferSourceNode[] = [];
  private sampleRate: number;

  constructor(sampleRate = 24000) {
    this.audioContext = new AudioContext();
    this.sampleRate = sampleRate;
  }

  async resume(): Promise<void> {
    if (this.audioContext.state !== "running") {
      await this.audioContext.resume();
    }
  }

  enqueuePcm16(bytes: Uint8Array): void {
    // Convert PCM16 bytes to Float32 for Web Audio API
    const floatData = this.pcm16BytesToFloat32(bytes);
    
    // Create audio buffer
    const buffer = this.audioContext.createBuffer(1, floatData.length, this.sampleRate);
    buffer.getChannelData(0).set(floatData);

    // Create and schedule source
    const src = this.audioContext.createBufferSource();
    src.buffer = buffer;
    src.connect(this.audioContext.destination);

    // Schedule seamless playback
    const now = this.audioContext.currentTime;
    if (this.nextStartTime < now) {
      this.nextStartTime = now;
    }
    const startAt = this.nextStartTime;
    this.nextStartTime += buffer.duration;

    // Track for cleanup
    src.onended = () => {
      this.sources = this.sources.filter((s) => s !== src);
    };
    this.sources.push(src);
    src.start(startAt);
  }

  private pcm16BytesToFloat32(bytes: Uint8Array): Float32Array {
    const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
    const sampleCount = Math.floor(bytes.byteLength / 2);
    const out = new Float32Array(sampleCount);
    for (let i = 0; i < sampleCount; i++) {
      const s = view.getInt16(i * 2, true);  // Little-endian
      out[i] = s / 0x8000;
    }
    return out;
  }

  stop(): void {
    for (const s of this.sources) {
      try {
        s.stop();
      } catch {
        /* ignore */
      }
    }
    this.sources = [];
    this.nextStartTime = 0;
  }
}
```

### Pattern B: AudioWorklet Playback

Better for handling barge-in (user interrupts assistant).

```javascript
// audio-playback-worklet.js
class AudioPlaybackWorklet extends AudioWorkletProcessor {
  constructor() {
    super();
    this.port.onmessage = this.handleMessage.bind(this);
    this.buffer = [];
  }

  handleMessage(event) {
    if (event.data === null) {
      // Clear buffer on barge-in
      this.buffer = [];
      return;
    }
    // Append Int16 samples
    this.buffer.push(...event.data);
  }

  process(inputs, outputs) {
    const output = outputs[0];
    const channel = output[0];

    if (this.buffer.length > channel.length) {
      const toProcess = this.buffer.slice(0, channel.length);
      this.buffer = this.buffer.slice(channel.length);
      channel.set(toProcess.map((v) => v / 32768));  // Int16 to Float32
    } else {
      channel.set(this.buffer.map((v) => v / 32768));
      this.buffer = [];
    }

    return true;
  }
}

registerProcessor("audio-playback-worklet", AudioPlaybackWorklet);
```

## PCM16 Conversion Utilities

```typescript
// Float32 to PCM16 (for sending to server)
export function floatTo16BitPCM(input: Float32Array): Int16Array {
  const output = new Int16Array(input.length);
  for (let i = 0; i < input.length; i++) {
    const s = Math.max(-1, Math.min(1, input[i] ?? 0));
    output[i] = s < 0 ? s * 0x8000 : s * 0x7fff;
  }
  return output;
}

// PCM16 bytes to Float32 (for playback)
export function pcm16BytesToFloat32LE(bytes: Uint8Array): Float32Array {
  const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
  const sampleCount = Math.floor(bytes.byteLength / 2);
  const out = new Float32Array(sampleCount);
  for (let i = 0; i < sampleCount; i++) {
    const s = view.getInt16(i * 2, true);  // Little-endian
    out[i] = s / 0x8000;
  }
  return out;
}

// Int16Array to Uint8Array (for sendAudio)
export function int16ToUint8(int16Array: Int16Array): Uint8Array {
  return new Uint8Array(int16Array.buffer);
}
```

## Integration with VoiceLiveSession

```typescript
import { VoiceLiveClient } from "@azure/ai-voicelive";
import { DefaultAzureCredential } from "@azure/identity";

const client = new VoiceLiveClient(endpoint, new DefaultAzureCredential());
const session = await client.startSession("gpt-4o-mini-realtime-preview");

// Configure audio formats
await session.updateSession({
  modalities: ["audio", "text"],
  inputAudioFormat: "pcm16",
  outputAudioFormat: "pcm16",
  turnDetection: {
    type: "server_vad",
    threshold: 0.5,
    silenceDurationMs: 500,
  },
});

// Initialize audio capture and playback
const player = new Pcm16Player(24000);
const capture = new AudioWorkletCapture((data) => {
  session.sendAudio(data);
});

// Handle audio output from assistant
const subscription = session.subscribe({
  onResponseAudioDelta: async (event) => {
    player.enqueuePcm16(event.delta);
  },
  onInputAudioBufferSpeechStarted: async () => {
    // User started speaking - stop playback (barge-in)
    player.stop();
  },
});

// Start capture
await capture.start();
await player.resume();

// Cleanup
async function cleanup() {
  capture.stop();
  player.stop();
  await subscription.close();
  await session.close();
}
```

## Best Practices

1. **Use AudioWorklet over ScriptProcessorNode** - Better performance and officially supported
2. **Disable browser audio processing** - Set `echoCancellation: false` when server handles VAD
3. **Handle barge-in** - Clear playback buffer when user starts speaking
4. **Match sample rates** - Ensure capture matches server expectation (usually 24kHz)
5. **Use transferable objects** - Pass `ArrayBuffer` with transfer list for efficiency
6. **Resume AudioContext on user gesture** - Browser policy requires user interaction

## See Also

- [Function Calling Reference](./function-calling.md)
- [SKILL.md](../SKILL.md) - Main skill documentation
- [Azure Voice Live Samples](https://github.com/microsoft-foundry/voicelive-samples)

````


### `references/function-calling.md`

````markdown
# Function Calling Reference

Tool definitions and function call handling patterns for Azure AI Voice Live using the @azure/ai-voicelive TypeScript SDK.

## Overview

Voice Live supports function calling (tools) that allow the voice assistant to execute actions and retrieve information during a conversation. This reference covers tool definitions, handling function call events, and sending function results.

## Tool Definition Schema

Tools are defined using JSON Schema in the session configuration:

```typescript
interface FunctionTool {
  type: "function";
  name: string;
  description?: string;
  parameters?: {
    type: "object";
    properties: Record<string, {
      type: string;
      description?: string;
      enum?: string[];
    }>;
    required?: string[];
  };
}
```

## Session Configuration with Tools

```typescript
import { VoiceLiveClient } from "@azure/ai-voicelive";
import { DefaultAzureCredential } from "@azure/identity";

const client = new VoiceLiveClient(endpoint, new DefaultAzureCredential());
const session = await client.startSession("gpt-4o-mini-realtime-preview");

await session.updateSession({
  modalities: ["audio", "text"],
  instructions: "You are a helpful assistant with access to weather data.",
  voice: {
    type: "azure-standard",
    name: "en-US-AvaNeural",
  },
  turnDetection: {
    type: "server_vad",
    threshold: 0.5,
    silenceDurationMs: 500,
  },
  tools: [
    {
      type: "function",
      name: "get_weather",
      description: "Get the current weather for a location",
      parameters: {
        type: "object",
        properties: {
          location: {
            type: "string",
            description: "City and state, e.g., San Francisco, CA",
          },
          unit: {
            type: "string",
            enum: ["celsius", "fahrenheit"],
            description: "Temperature unit",
          },
        },
        required: ["location"],
      },
    },
    {
      type: "function",
      name: "search_products",
      description: "Search for products in the catalog",
      parameters: {
        type: "object",
        properties: {
          query: {
            type: "string",
            description: "Search query",
          },
          category: {
            type: "string",
            enum: ["electronics", "clothing", "home", "sports"],
          },
          max_results: {
            type: "integer",
            description: "Maximum number of results to return",
          },
        },
        required: ["query"],
      },
    },
  ],
  toolChoice: "auto",  // "auto", "none", "required", or { type: "function", name: "..." }
});
```

## Handling Function Calls

### Basic Pattern

```typescript
const subscription = session.subscribe({
  onResponseFunctionCallArgumentsDone: async (event, context) => {
    console.log(`Function called: ${event.name}`);
    console.log(`Call ID: ${event.callId}`);
    console.log(`Arguments: ${event.arguments}`);

    // Parse arguments
    const args = JSON.parse(event.arguments);

    // Execute function based on name
    let result: string;
    
    switch (event.name) {
      case "get_weather":
        result = await getWeather(args.location, args.unit ?? "celsius");
        break;
      case "search_products":
        result = await searchProducts(args.query, args.category, args.max_results);
        break;
      default:
        result = JSON.stringify({ error: `Unknown function: ${event.name}` });
    }

    // Send function result back
    await session.addConversationItem({
      type: "function_call_output",
      callId: event.callId,
      output: result,
    });

    // Trigger response generation
    await session.sendEvent({ type: "response.create" });
  },
});
```

### Complete Example with Multiple Tools

```typescript
import { VoiceLiveClient, VoiceLiveSession } from "@azure/ai-voicelive";
import { DefaultAzureCredential } from "@azure/identity";

// Function implementations
async function getWeather(location: string, unit: string): Promise<string> {
  // In production, call actual weather API
  const temp = unit === "celsius" ? "22°C" : "72°F";
  return JSON.stringify({
    location,
    temperature: temp,
    conditions: "Partly cloudy",
    humidity: "45%",
  });
}

async function searchProducts(
  query: string,
  category?: string,
  maxResults = 5
): Promise<string> {
  // In production, query product database
  const products = [
    { id: "1", name: `${query} - Premium`, price: 99.99 },
    { id: "2", name: `${query} - Standard`, price: 49.99 },
  ];
  return JSON.stringify({
    query,
    category: category ?? "all",
    results: products.slice(0, maxResults),
  });
}

async function bookAppointment(
  date: string,
  time: string,
  service: string
): Promise<string> {
  // In production, book in calendar system
  return JSON.stringify({
    success: true,
    confirmation: `APT-${Date.now()}`,
    date,
    time,
    service,
  });
}

// Function dispatcher
async function handleFunctionCall(
  name: string,
  args: Record<string, unknown>
): Promise<string> {
  switch (name) {
    case "get_weather":
      return getWeather(
        args.location as string,
        (args.unit as string) ?? "celsius"
      );
    case "search_products":
      return searchProducts(
        args.query as string,
        args.category as string | undefined,
        args.max_results as number | undefined
      );
    case "book_appointment":
      return bookAppointment(
        args.date as string,
        args.time as string,
        args.service as string
      );
    default:
      return JSON.stringify({ error: `Unknown function: ${name}` });
  }
}

// Main voice assistant
async function startAssistant() {
  const client = new VoiceLiveClient(endpoint, new DefaultAzureCredential());
  const session = await client.startSession("gpt-4o-mini-realtime-preview");

  await session.updateSession({
    modalities: ["audio", "text"],
    instructions: `You are a helpful assistant that can:
      - Get weather information
      - Search for products
      - Book appointments
      Always confirm details with the user before booking.`,
    tools: [
      {
        type: "function",
        name: "get_weather",
        description: "Get current weather",
        parameters: {
          type: "object",
          properties: {
            location: { type: "string", description: "City name" },
            unit: { type: "string", enum: ["celsius", "fahrenheit"] },
          },
          required: ["location"],
        },
      },
      {
        type: "function",
        name: "search_products",
        description: "Search product catalog",
        parameters: {
          type: "object",
          properties: {
            query: { type: "string" },
            category: { type: "string", enum: ["electronics", "clothing", "home"] },
            max_results: { type: "integer" },
          },
          required: ["query"],
        },
      },
      {
        type: "function",
        name: "book_appointment",
        description: "Book an appointment",
        parameters: {
          type: "object",
          properties: {
            date: { type: "string", description: "Date in YYYY-MM-DD format" },
            time: { type: "string", description: "Time in HH:MM format" },
            service: { type: "string", description: "Type of service" },
          },
          required: ["date", "time", "service"],
        },
      },
    ],
    toolChoice: "auto",
  });

  const subscription = session.subscribe({
    onResponseFunctionCallArgumentsDone: async (event, context) => {
      console.log(`Executing: ${event.name}`);
      
      try {
        const args = JSON.parse(event.arguments);
        const result = await handleFunctionCall(event.name, args);

        await session.addConversationItem({
          type: "function_call_output",
          callId: event.callId,
          output: result,
        });

        await session.sendEvent({ type: "response.create" });
      } catch (error) {
        // Send error as function output
        await session.addConversationItem({
          type: "function_call_output",
          callId: event.callId,
          output: JSON.stringify({ error: String(error) }),
        });
        await session.sendEvent({ type: "response.create" });
      }
    },

    onResponseTextDelta: async (event) => {
      process.stdout.write(event.delta);
    },

    onError: async (args) => {
      console.error("Session error:", args.error);
    },
  });

  return { session, subscription };
}
```

## Streaming Function Arguments

For long-running function calls, you can process arguments as they stream in:

```typescript
const pendingCalls = new Map<string, { name: string; arguments: string }>();

const subscription = session.subscribe({
  // Arguments arrive in chunks
  onResponseFunctionCallArgumentsDelta: async (event, context) => {
    const callId = event.callId;
    
    if (!pendingCalls.has(callId)) {
      pendingCalls.set(callId, { name: event.name ?? "", arguments: "" });
    }
    
    const pending = pendingCalls.get(callId)!;
    if (event.name) {
      pending.name = event.name;
    }
    if (event.delta) {
      pending.arguments += event.delta;
    }
  },

  // All arguments received
  onResponseFunctionCallArgumentsDone: async (event, context) => {
    const callId = event.callId;
    const pending = pendingCalls.get(callId);
    pendingCalls.delete(callId);

    if (!pending) {
      console.error("No pending call found for:", callId);
      return;
    }

    // Now process the complete function call
    const args = JSON.parse(pending.arguments || event.arguments);
    const result = await handleFunctionCall(pending.name || event.name, args);

    await session.addConversationItem({
      type: "function_call_output",
      callId,
      output: result,
    });

    await session.sendEvent({ type: "response.create" });
  },
});
```

## Parallel Function Calls

Voice Live may invoke multiple functions in parallel. Handle them all before triggering response:

```typescript
const pendingCalls = new Map<string, Promise<string>>();
let pendingCount = 0;

const subscription = session.subscribe({
  onResponseFunctionCallArgumentsDone: async (event, context) => {
    pendingCount++;
    
    // Start function execution (don't await)
    const resultPromise = handleFunctionCall(
      event.name,
      JSON.parse(event.arguments)
    );
    pendingCalls.set(event.callId, resultPromise);
  },

  onResponseDone: async (event, context) => {
    // Check if there are pending function calls
    if (pendingCalls.size === 0) return;

    // Wait for all functions to complete
    for (const [callId, promise] of pendingCalls) {
      const result = await promise;
      await session.addConversationItem({
        type: "function_call_output",
        callId,
        output: result,
      });
    }

    pendingCalls.clear();
    
    // Trigger response generation after all results sent
    await session.sendEvent({ type: "response.create" });
  },
});
```

## Error Handling

```typescript
const subscription = session.subscribe({
  onResponseFunctionCallArgumentsDone: async (event, context) => {
    try {
      const args = JSON.parse(event.arguments);
      const result = await handleFunctionCall(event.name, args);

      await session.addConversationItem({
        type: "function_call_output",
        callId: event.callId,
        output: result,
      });
    } catch (error) {
      // Send error message so assistant can respond appropriately
      const errorMessage = error instanceof Error ? error.message : String(error);
      
      await session.addConversationItem({
        type: "function_call_output",
        callId: event.callId,
        output: JSON.stringify({
          error: true,
          message: `Function execution failed: ${errorMessage}`,
        }),
      });
    }

    await session.sendEvent({ type: "response.create" });
  },
});
```

## Tool Choice Options

Control how the model selects tools:

```typescript
// Auto - model decides when to use tools
await session.updateSession({ toolChoice: "auto" });

// None - disable tool usage
await session.updateSession({ toolChoice: "none" });

// Required - model must use a tool
await session.updateSession({ toolChoice: "required" });

// Force specific function
await session.updateSession({
  toolChoice: {
    type: "function",
    name: "get_weather",
  },
});
```

## Conversation Item Types

```typescript
// Add user message
await session.addConversationItem({
  type: "message",
  role: "user",
  content: [{ type: "text", text: "What's the weather in Seattle?" }],
});

// Add function call output
await session.addConversationItem({
  type: "function_call_output",
  callId: "call_abc123",
  output: JSON.stringify({ temperature: "72°F", conditions: "Sunny" }),
});

// Trigger response
await session.sendEvent({ type: "response.create" });
```

## Best Practices

1. **Return JSON from functions** - Structured data is easier for the model to interpret
2. **Include error information** - Return `{ error: true, message: "..." }` on failure
3. **Handle timeouts** - Set reasonable timeouts for external API calls
4. **Validate arguments** - Check required fields before executing functions
5. **Use descriptive names** - Function and parameter names help the model understand usage
6. **Provide examples in description** - E.g., "City and state, e.g., San Francisco, CA"
7. **Handle parallel calls** - Don't assume functions are called sequentially

## See Also

- [Audio Streaming Reference](./audio-streaming.md)
- [SKILL.md](../SKILL.md) - Main skill documentation
- [Azure Voice Live API Reference](https://learn.microsoft.com/javascript/api/@azure/ai-voicelive)

````