WebCodecs

1. Introduction

The API defined in this document provides efficient access to built-in (software and hardware) media encoders and decoders for encoding and decoding media. There are many Web APIs that use media codecs internally to support like HTMLMediaElement, WebAudio, MediaRecorder and WebRTC. Inspite of the wide spread usage, there has been a lack of support for configuring the media codecs. As a consequence, many web applications have resorted to implementing media codecs in JavaScript or WebAssembly. This results in reduced power efficiency, reduced performance and increased bandwidth to download a media codec already present in the browser. A comprehensive list of applicable use cases, and code examples can be found in [WEBCODECS-USECASES] and [WEBCODECS-EXAMPLES] explainer documents.

2. Scope

3. Background

This section is non-normative.

4. Use Cases

This section is non-normative.

This section provides a collection of use cases and usage scenarions for web pages and applications using WebCodecs.

• Low latency live streaming (< 100 ms delay).

• Cloud gaming: WebRTC and MSE are not great for cloud gaming. WebSocket is also not great for gaming. WebRTC is hard to use in a server-client architecture and does not provide a lot of knobs to control buffering, decoding and rendering which are all controlled by browser. Cloud gaming need low latency provided by WebTransport and more web developer control provided by WebCodecs. Details are described at [CLOUD-GAMING-WEBCODECS].

• Live stream uploading.

• Non-realtime encoding/decoding/transcoding, such as for local file editing.

• Advanced Real-time Communications.

  • End-to-end encryption.

  • Control over buffer behavior.

  • Spatial and temporal scalability.

• Re-encoding multiple input media streams in order to merge many encoded media. streams into one encoded media stream.

5. Security and Privacy Considerations

6. Model

An EncodedAudioChunks and EncodedVideoChunks contain codec-specific encoded media bytes. An EncodedVideoChunk contains a single encoded video frame along with metadata related to the frame, for example timestamp.

AudioPacket contains decoded audio data. It will provide an AudioBuffer for rendering via AudioWorklet.

A VideoFrame contains decoded video data. It can be drawn into Canvas with drawImage or rendered into WebGL texture with texImage2D.

Support VideoFrame creation from yuv data? See WICG#45

An AudioEncoder encodes AudioPackets to produce EncodedAudioChunks.

A VideoEncoder encodes VideoFrames to produce EncodedVideoChunks.

An AudioDecoder decodes EncodedAudioChunks to produce AudioPackets.

A VideoDecoder decodes EncodedVideoChunks to produce VideoFrames.

WebCodecs API also has mechanisms to import content referenced through a valid MediaStreamTrack, for example from getUserMedia.

The term platform decoder refers to the platform interfaces with which the user agent interacts to obtain a decoded VideoFrame. The platform decoder can be defined by the underlying platform (e.g native media framework).

The term platform encoder refers to the platform interfaces with which the user agent interacts to encode a VideoFrame. The platform encoder can be defined by the underlying platform (e.g native media framework).

7. WebCodecs API

7.1. VideoFrame interface

dictionary VideoFrameInit {
  unsigned long long timestamp;  // microseconds
  unsigned long long? duration;  // microseconds
};

[Exposed=(Window)]
interface VideoFrame {
    constructor(VideoFrameInit init, ImageBitmap source);
    void release();
    [NewObject] Promise<ImageBitmap> createImageBitmap(optional ImageBitmapOptions options = {});
    readonly attribute unsigned long long timestamp;  // microseconds
    readonly attribute unsigned long long? duration;  // microseconds
    readonly attribute unsigned long codedWidth;
    readonly attribute unsigned long codedHeight;

    readonly attribute unsigned long visibleWidth;
    readonly attribute unsigned long visibleHeight;
};

Instances of VideoFrame are created with the internal slots described in the following table:

The timestamp attribute in VideoFrame object represents the sampling instant of the first data of the VideoFrame in microseconds from a fixed origin. Initial value should be specified by timestamp during VideoFrame construction.

The duration is an attribute in VideoFrame object that represents the time interval for which the video composition should render the composed VideoFrame in microseconds.

The codedWidth attribute in VideoFrame object denotes the number of pixel samples stored horizontally for each frame.

The codedHeight attribute in VideoFrame object denotes the number of pixel samples stored vertically for each frame.

The visibleWidth attribute in VideoFrame object denotes the number of pixel samples horizontally which should be visible to the user.

The visibleHeight attribute in VideoFrame object denotes the number of pixel samples vertically which should be visible to the user.

Note: An image’s clean aperture is a region of video free from transition artifacts caused by the encoding of the signal. This is the region of video that should be displayed. visibleWidth and visibleHeight denote the frame’s clean aperture region. The clean aperture is usually in the center of production aperture which might contain some details along the edges of the image. The codedWidth and codedHeight constitute the production aperture of the image.

How to express encoded size vs. visible size vs. natural size WICG#26

7.1.1. Create VideoFrame

7.1.2. VideoFrame.createImageBitmap() method

7.1.3. VideoFrame.release() method

[ttoivone] From the Javascript perspective, release is not needed but it could just clear all references to a VideoFrame object and let garbage collector to release the memory. Should guidelines be given here when the function needs to be called explicitly?

7.2. EncodedVideoChunk interface

enum EncodedVideoChunkType {
  "key",
  "delta",
};
interface EncodedVideoChunk {
  constructor(EncodedVideoChunkType chunk_type, unsigned long long chunk_timestamp, BufferSource chunk_data);
  constructor(EncodedVideoChunkType chunk_type, unsigned long long chunk_timestamp, unsigned long long chunk_duration, BufferSource chunk_data);
  readonly attribute EncodedVideoChunkType type;
  readonly attribute unsigned long long timestamp;  // microseconds
  readonly attribute unsigned long long? duration;  // microseconds
  readonly attribute ArrayBuffer data;
};

The type attribute in EncodedVideoChunk is set to key if the encoded video frame stored in the chunk is a key frame (ie. the frame can be decoded independently without referring to other frames) or otherwise it is set to delta.

The timestamp attribute in EncodedVideoChunk object represents the sampling instant of the data in the EncodedVideoChunk in microseconds from a fixed origin.

The duration is an attribute in EncodedVideoChunk object that represents the time duration for which the video composition should render the composed EncodedVideoChunk in microseconds.

The data attribute in EncodedVideoChunk stores the video frame in encoded form.

7.2.1. Create EncodedVideoChunk

7.3. AudioPacket interface

7.4. EncodedAudioChunk interface

7.5. WebCodecs callbacks

callback WebCodecsErrorCallback = void (DOMException error);
callback VideoFrameOutputCallback = void (VideoFrame output);
callback VideoEncoderOutputCallback = void (EncodedVideoChunk chunk);

7.6. VideoDecoder interface

[Exposed=(Window)]
interface VideoDecoder {
    constructor(VideoDecoderInit init);

    Promise<void> configure(EncodedVideoConfig config);
    Promise<void> decode(EncodedVideoChunk chunk);
    Promise<void> flush();
    Promise<void> reset();

    readonly attribute long decodeQueueSize;
    readonly attribute long decodeProcessingCount;
};

dictionary VideoDecoderInit {
  VideoFrameOutputCallback output;
  WebCodecsErrorCallback error;
};

dictionary EncodedVideoConfig {
  required DOMString codec;
  BufferSource description;
  double sampleAspect;
};

A VideoDecoder object processes a queue of configure, decode, and flush requests. Requests are taken from the queue sequentially but may be processed concurrently. A VideoDecoder object has an associated platform decoder.

7.6.1. VideoDecoder.decodeQueueSize

The decodeQueueSize attribute in VideoDecoder object denotes the number of queued decode requests, excluding those that are already being processed or have finished processing. Applications can minimize underflow by enqueueing decoding requests until decodeQueueSize is sufficiently large.

7.6.2. VideoDecoder.decodeProcessingCount

The decodeProcessingCount attribute in VideoDecoder object denotes the number of decode requests currently being processed. Applications can minimize resource consumption and decode latency by enqueueing decode requests only when decodeQueueSize and decodeProcessingCount are small.

7.6.3. VideoDecoder Callbacks

The VideoDecoderOutputCallback, denoted by output, is for emitting VideoFrames. The WebCodecsErrorCallback, denoted by error, is for emitting decode errors.

7.6.4. VideoDecoder internal slots

Instances of VideoDecoder are created with the internal slots described in the following table:

7.6.5. Create VideoDecoder

7.6.6. VideoDecoder.configure() method

Note: After the configure() call, the decoder is in the newly initialized state so the next chunk to be decoded must be a keyframe.

7.6.7. VideoDecoder.decode() method

Note: get backpressure idea from decodeQueueSize) to know if [[platform_decoder]] can accept new work right away with decodeProcessingCount.

7.6.8. VideoDecoder.flush() method

output

p, a Promise object.

7.6.9. VideoDecoder.reset() method

output

p, a Promise object.

Note: After the reset() call, the decoder is in the newly initialized state so the next chunk to be decoded must be a keyframe.

7.7. VideoEncoder interface

dictionary VideoEncoderTuneOptions {
  unsigned long long bitrate;
  double framerate;
  required unsigned long width;
  required unsigned long height;
};

dictionary VideoEncoderInit {
  required DOMString codec;
  DOMString profile;
  required VideoEncoderTuneOptions tuneOptions;
  required VideoEncoderOutputCallback output;
  WebCodecsErrorCallback error;
};

dictionary VideoEncoderEncodeOptions {
  boolean? keyFrame;
};

[Exposed=(Window)]
interface VideoEncoder {
  constructor();
  Promise<void> configure(VideoEncoderInit init);
  Promise<void> encode(VideoFrame frame, optional VideoEncoderEncodeOptions options);
  Promise<void> tune(VideoEncoderTuneOptions options);
  Promise<void> flush();
  Promise<void> close();
};

A VideoEncoder object processes a queue of configure, encode, tuning to new parameters and flush requests. Requests are taken from the queue sequentially but may be processed concurrently. A VideoEncoder object has an associated platform video encoder.

7.7.1. VideoEncoder internal slots

Instances of VideoEncoder are created with the internal slots described in the following table:

7.7.2. Create a VideoEncoder

7.7.3. VideoEncoder.configure() method

7.7.4. VideoEncoder.encode() method

7.7.5. VideoEncoder.tune() method

7.7.6. VideoEncoder.flush() method

7.7.7. VideoEncoder.close() method

7.8. AudioDecoder interface

7.9. AudioEncoder interface

8. Examples

// App provides stream of encoded chunks to decoder.
function streamEncodedChunks(decodeCallback) { ... }

// The document contains a canvas for displaying VideoFrames.
const canvasElement = document.getElementById("canvas");
const canvasContext = canvasElement.getContext('bitmaprenderer');

function paintFrameToCanvas(videoFrame) {
  // Paint every video frame ASAP for lowest latency.
  canvasContext.transferFromImageBitmap(videoFrame.transferToImageBitmap());
}

const videoDecoder = new VideoDecoder({
  output: paintFrameToCanvas,
  error: console.error("Decode Error")
});

videoDecoder.configure({codec: 'vp8'}).then(() => {
  // The app fetches VP8 chunks, feeding each chunk to the decode
  // callback as fast as possible. Real apps must also monitor
  // decoder backpressure to ensure the decoder is keeping up.
  streamEncodedChunks(videoDecoder.decode.bind(videoDecoder));
}).catch(() => {
  // App provides fallback logic when config not supported.
  ...
});

// App demuxes (decontainerizes) input and makes repeated calls to the provided
// callbacks to feed the decoders.
function streamEncodedChunks(decodeAudioCallback, decodeVideoCallback) { ... }

// App provides a way to demux  and mux (containerize) media.
function muxAudio(encodedChunk) { ... }
function muxVideo(encodedChunk) { ... }

// The app provides error handling (e.g. shutdown w/ UI message)
function onCodecError(error) { ... }

// Returns an object { audioEncoder, videoEncoder }.
// Encoded outputs sent immediately to app provided muxer.
async function buildAndConfigureEncoders() {
  // Build encoders.
  let audioEncoder = new AudioEncoder({ output: muxAudio, error: onCodecError });
  let videoEncoder = new VideoEncoder({ output: muxVideo, error: onCodecError });

  // Configure and reset if not supported. More sophisticated fallback recommended.
  try {
    await audioEncoder.configure({ codec: 'opus', ... });
  } catch (error) {
    audioEncoder = null;
  }
  try {
    await videoEncoder.configure({ codec : 'vp8', ... });
  } catch (error) {
    videoEncoder = null;
  }

  return {audioEncoder, videoEncoder };
}

// Returns an object { audioDecoder, videoDecoder }.
// Decoded outputs sent immediately to the coresponding encoder for re-encoding.
async function buildAndConfigureDecoders(audioEncoder, videoEncoder) {
  // Bind encode callbacks.
  const reEncodeAudio = audioEncoder.encode.bind(audioEncoder);
  const reEncodeVideo = videoEncode.encode.bind(videoEncoder);

  // Build decoders.
  const audioDecoder = new AudioDecoder({ output: reEncodeAudio, error: onCodecError });
  const videoDecoder = new VideoDecoder({ output: reEncodeVideo, error: onCodecError });

  // Configure and reset if not supported. More sophisticated fallback recommended.
  try {
    await audioDecoder.configure({ codec: 'aac', ... });
  } catch (error) {
    audioDecoder = null;
  }
  try {
    await videoDecoder.configure({ codec : 'avc1.42001e', ... });
  } catch (error) {
    videoDecoder = null;
  }

  return { audioDecoder, videoDecoder};
}

// Setup encoders.
let { audioEncoder, videoEncoder } = await buildAndConfigureEncoders();

// App handles unsupported configuration.
if (audioEncoder == null || videoEncoder == null)
  return;

// Setup decoders. Provide encoders to receive decoded output.
let { audioDecoder, videoDecoder } = await buildAndConfigureDecoders(audioEncoder, videoEncoder);

// App handles unsupported configuration.
if (audioDecoder == null || videoDecoder == null)
  return;

// Start streaming encoded chunks to the decoders, repeatedly calling
// the provided callbacks for each chunk.
// Decoded output will be fed to encoders for re-encoding.
// Encoded output will be fed to muxer.
streamEncodedChunks(audioDecoder.decode.bind(audioDecoder),
          videoDecoder.decode.bind(videoDecoder));

  videoEncoder.configure({
  codec : 'vp9',
  tuning: {
    bitrate: 1_000_000,
    framerate: 24,
    width: 1024,
    height: 768
  }
  // Two spatial layers with two temporal layers each
  layers: [{
      // Quarter size base layer
      id: 'p0',
      temporalSlots: [0],
      scaleDownBy: 2,
      dependsOn: ['p0'],
    }, {
      id: 'p1'
      temporalSlots: [1],
      scaleDownBy: 2,
      dependsOn: ['p0'],
    }, {
      id: 's0',
      temporalSlots: [0],
      dependsOn: ['p0', 's0'],
    }, {
      id: 's1',
      temporalSlots: [1],
      dependsOn: ['p1', 's0', 's1'],
    }],
});

9. Acknowledgements

The following people have greatly contributed to this specification through extensive discussions on GitHub: