Custom Audio Handling - Cognigy Documentation

Default Behavior

By default, the SDK:

1. Creates an internal Audio element with autoplay enabled and volume set to 1.0.
2. When a remote audio stream arrives via the WebRTC peer connection, the SDK attaches it to the Audio element and starts playback automatically.
3. When the call ends, the SDK stops playback, releases the stream, and emits the audioEnded event.

No manual audio setup is required for basic call functionality.

Browser autoplay policies may block audio playback if the call is not initiated by a user gesture, for example, a button click. Always start calls in response to user interaction.

Accessing the Raw Audio Stream

To receive the raw MediaStream for advanced audio processing, enable the captureAudio option and subscribe to the event:

const client = await createWebRTCClient({
  endpointUrl: 'https://your-cognigy-environment.com/token',
  userId: 'user-123',
  captureAudio: true,
});

client.on('captureAudio', (stream) => {
  // The stream contains the remote audio track
  console.log('Audio stream received:', stream.getAudioTracks());
});

The captureAudio event emits in addition to the SDK’s automatic audio playback. You don’t need to handle playback yourself — the SDK still plays audio through its internal Audio element.

Volume Control

Use the Web Audio API to add programmatic volume control on top of the captured stream:

let gainNode: GainNode;
let audioContext: AudioContext;

client.on('captureAudio', (stream) => {
  audioContext = new AudioContext();
  const source = audioContext.createMediaStreamSource(stream);

gainNode = audioContext.createGain();
  gainNode.gain.value = 1.0; // Full volume

source.connect(gainNode);
  // Don't connect to audioContext.destination — the SDK already handles playback.
  // Use gainNode only if you need to route audio to a custom destination.
});

// Adjust volume (0.0 = silent, 1.0 = full)
function setVolume(level: number) {
  if (gainNode) {
    gainNode.gain.value = Math.max(0, Math.min(1, level));
  }
}

The example above does not connect to audioContext.destination because the SDK already plays audio through its internal Audio element. If you connect to audioContext.destination, you will hear audio twice. If you want to fully replace the SDK’s audio playback with your own Web Audio pipeline, mute the internal audio element first.

Audio Visualization

Create a real-time audio visualizer using the AnalyserNode:

client.on('captureAudio', (stream) => {
  const audioContext = new AudioContext();
  const source = audioContext.createMediaStreamSource(stream);
  const analyser = audioContext.createAnalyser();
  analyser.fftSize = 256;

source.connect(analyser);
  // Don't connect to audioContext.destination — the SDK already plays audio
  // Connecting again would cause double playback

const dataArray = new Uint8Array(analyser.frequencyBinCount);

function draw() {
    requestAnimationFrame(draw);
    analyser.getByteFrequencyData(dataArray);

// Use dataArray values to render a visualization
    const average = dataArray.reduce((a, b) => a + b, 0) / dataArray.length;
    console.log('Audio level:', average);
  }

draw();
});

When using the captureAudio stream for visualization, avoid connecting it to audioContext.destination since the SDK already handles playback. Doing so would cause the audio to play twice.

Audio Recording

Record the remote audio stream using the MediaRecorder API:

let mediaRecorder: MediaRecorder;
const recordedChunks: Blob[] = [];

client.on('captureAudio', (stream) => {
  // Record audio — playback is handled automatically by the SDK
  mediaRecorder = new MediaRecorder(stream, { mimeType: 'audio/webm' });
  mediaRecorder.ondataavailable = (event) => {
    if (event.data.size > 0) {
      recordedChunks.push(event.data);
    }
  };
  mediaRecorder.start();
});

client.on('audioEnded', () => {
  if (mediaRecorder && mediaRecorder.state !== 'inactive') {
    mediaRecorder.stop();
  }
});

// Save recording
function saveRecording() {
  const blob = new Blob(recordedChunks, { type: 'audio/webm' });
  const url = URL.createObjectURL(blob);
  const a = document.createElement('a');
  a.href = url;
  a.download = 'call-recording.webm';
  a.click();
}

Recording calls may be subject to legal requirements such as consent laws. Ensure your application complies with applicable regulations before recording audio.