Video Codecs

A PeerConnectionFactory decides which video codecs it can send and receive through a video encoder factory and a video decoder factory. Unless told otherwise it uses the codecs built into the library. This guide explains how to replace or extend them: to add a codec WebRTC does not have, to use a hardware encoder reached through another library, or to limit the codecs a factory offers.

The API lives in the dev.onvoid.webrtc.media.video.codec package.

Built-in Codecs

DefaultVideoEncoderFactory and DefaultVideoDecoderFactory provide the codecs built into the library. On Windows and Linux these are VP8, VP9, AV1 and H.264 in software; on macOS they are those of WebRTC's default factories, with H.264 through VideoToolbox. Ask a factory for the exact list:

for (VideoCodecInfo codec : new DefaultVideoEncoderFactory().getSupportedCodecs()) {
    System.out.println(codec.getName() + " " + codec.getParameters());
}

Hardware Encoding

HardwareVideoEncoderFactory offers the same codecs as DefaultVideoEncoderFactory, but encodes on the GPU where the platform supports it:

PeerConnectionFactory factory = PeerConnectionFactory.builder()
        .setVideoEncoderFactory(new HardwareVideoEncoderFactory())
        .build();

Platform	H.264	AV1
Windows	NVENC on NVIDIA GPUs, otherwise the Media Foundation encoder of the GPU driver (AMD, Intel)	NVENC on NVIDIA RTX 40 and newer, otherwise Media Foundation on GPUs that encode AV1 (e.g. AMD RDNA3 and newer, Intel Arc)
Linux	NVENC on NVIDIA GPUs, otherwise the VA-API encoder of the GPU driver (Intel, AMD)	NVENC on NVIDIA RTX 40 and newer
macOS	VideoToolbox, as with DefaultVideoEncoderFactory	Software

Hardware encoders produce a single layer. A stream that asks for SVC, such as an AV1 stream with a scalability mode like L1T3, is encoded in software.

NVENC needs an NVIDIA driver of version 522 or newer on Windows, 520 or newer on Linux. VA-API needs libva 2 and a driver that encodes H.264, such as Intel's intel-media-va-driver (iHD) or Mesa's mesa-va-drivers for AMD, and access to a render node in /dev/dri. All of them are loaded at run time, so nothing needs to be installed on machines without them.

The hardware encoders take over H.264 Constrained Baseline and Baseline with packetization mode 1, and AV1 profile 0, formats the software encoders offer too, so encoding in hardware never changes what is negotiated. When a hardware encoder fails to start, for example because the GPU has no encoder sessions left, or fails while encoding, the stream switches to the next encoder in line (e.g. from NVENC to Media Foundation on Windows), and finally to the software encoder, and continues with a key frame. On a machine without a hardware encoder, the factory encodes like DefaultVideoEncoderFactory.

Which encoder a stream uses shows in the encoderImplementation statistic of its outbound-rtp stats, e.g. NVENC (NVIDIA GeForce RTX 4070), MediaFoundation (AMDav1Encoder), VA-API (Intel iHD driver ...), OpenH264 or libaom.

Hardware Decoding

HardwareVideoDecoderFactory does the same for decoding:

PeerConnectionFactory factory = PeerConnectionFactory.builder()
        .setVideoEncoderFactory(new HardwareVideoEncoderFactory())
        .setVideoDecoderFactory(new HardwareVideoDecoderFactory())
        .build();

Platform	Hardware decoding
Windows	H.264 and AV1, on GPUs that decode them, through the Media Foundation decoders of Windows on Direct3D 11 (DXVA)
Linux	Not yet; decoding is in software
macOS	VideoToolbox, as with DefaultVideoDecoderFactory

AV1 on Windows needs the AV1 Video Extension, which Windows 11 includes. Decoded frames are copied from GPU memory back to system memory, where WebRTC's frames are, so hardware decoding pays off mostly at high resolutions and with many streams; at low resolutions WebRTC's software decoders are about as cheap. A hardware decoder that fails, or turns out to decode in software, is replaced by the software decoder, which starts with the next key frame.

Which decoder a stream uses shows in the decoderImplementation statistic of its inbound-rtp stats, e.g. MediaFoundation (Microsoft H264 Video Decoder MFT) or MediaFoundation (AV1VideoExtension).

Native Codecs

The encoders and decoders these factories create are NativeVideoEncoders and NativeVideoDecoders. They are placeholders that make WebRTC create the built-in codec, which then runs entirely inside WebRTC, so their methods are not to be called from Java.

Setting the Factories

The codec factories are set with PeerConnectionFactory.builder(), which also takes everything the constructors do:

PeerConnectionFactory factory = PeerConnectionFactory.builder()
        .setAudioDeviceModule(audioModule)
        .setVideoEncoderFactory(new MyEncoderFactory())
        .setVideoDecoderFactory(new MyDecoderFactory())
        .build();

A factory that is not set is the built-in one. The supported codecs are asked for once, when the PeerConnectionFactory is built; an exception thrown there fails build(). Encoders and decoders are created later, from WebRTC threads, one per stream.

Both ends have to agree on a codec, so a codec added on the sending side also has to be offered by the decoder factory of the receiving side.

Adding a Codec

A factory of one's own usually hands out its own codec next to the built-in ones, by delegating to a default factory:

class MyEncoderFactory implements VideoEncoderFactory {

    private final DefaultVideoEncoderFactory builtIn = new DefaultVideoEncoderFactory();

    @Override
    public List<VideoCodecInfo> getSupportedCodecs() {
        List<VideoCodecInfo> codecs = new ArrayList<>();
        codecs.add(new VideoCodecInfo("X-MYCODEC"));
        codecs.addAll(builtIn.getSupportedCodecs());
        return codecs;
    }

    @Override
    public VideoEncoder createEncoder(VideoCodecInfo info) {
        if (info.getName().equalsIgnoreCase("X-MYCODEC")) {
            return new MyEncoder();
        }
        return builtIn.createEncoder(info);
    }
}

The list is in order of preference. A codec name WebRTC does not know, such as X-MYCODEC, is negotiated like any other and sent with the generic RTP packetization. RTCRtpTransceiver.setCodecPreferences() picks among the negotiated codecs per transceiver.

Implementing an Encoder

WebRTC calls initEncode(), then encode() for each frame and setRates() whenever the target bitrate changes, and finally release(). It may initialize a released encoder again, possibly with another frame size. These calls come from one WebRTC thread at a time.

class MyEncoder implements VideoEncoder {

    private Callback callback;

    @Override
    public VideoCodecStatus initEncode(Settings settings, Callback callback) {
        this.callback = callback;
        // Set up the encoder for settings.width x settings.height at
        // settings.startBitrateKbps.
        return VideoCodecStatus.OK;
    }

    @Override
    public VideoCodecStatus encode(VideoFrame frame, EncodeInfo info) {
        I420Buffer pixels = frame.buffer.toI420();
        ByteBuffer payload = encodeSomehow(pixels, info.isKeyFrameRequested());

        callback.onEncodedFrame(EncodedImage.builder()
                .setBuffer(payload)
                .setEncodedWidth(pixels.getWidth())
                .setEncodedHeight(pixels.getHeight())
                .setCaptureTimeNs(frame.timestampNs)
                .setFrameType(info.isKeyFrameRequested()
                        ? EncodedImage.FrameType.KEY
                        : EncodedImage.FrameType.DELTA)
                .build());

        return VideoCodecStatus.OK;
    }

    @Override
    public VideoCodecStatus setRates(RateControlParameters parameters) {
        // Aim for parameters.bitrate.getSum() bps at parameters.framerateFps.
        return VideoCodecStatus.OK;
    }

    @Override
    public VideoCodecStatus release() {
        return VideoCodecStatus.OK;
    }
}

Things to keep in mind:

• The capture time of an EncodedImage must be the timestampNs of the frame it encodes. That is how WebRTC matches output to input; an image with any other capture time is dropped.
• The frame passed to encode() is valid only until the method returns. An encoder that works asynchronously calls frame.retain() and later frame.release(), and hands its output to the callback from its own thread, in encoding order. The pixels must not be changed, since other consumers may share them.
• onEncodedFrame() copies the payload, so the buffer may be reused once it returns. Frames handed over after release() are ignored.
• getScalingSettings(), getEncoderInfo(), getResolutionBitrateLimits(), getImplementationName() and isHardwareEncoder() have defaults and may be overridden. The default scaling settings use WebRTC's quantizer thresholds for VP8, VP9 and H.264; a custom codec without a quantizer should return ScalingSettings.OFF.

Implementing a Decoder

A decoder follows the same pattern with initDecode(), decode() and release():

class MyDecoder implements VideoDecoder {

    private Callback callback;

    @Override
    public VideoCodecStatus initDecode(Settings settings, Callback callback) {
        this.callback = callback;
        return VideoCodecStatus.OK;
    }

    @Override
    public VideoCodecStatus decode(EncodedImage image) {
        NativeI420Buffer buffer = decodeSomehow(image.getBuffer());
        VideoFrame frame = new VideoFrame(buffer, image.getCaptureTimeNs());

        callback.onDecodedFrame(frame, null, null);
        frame.release();

        return VideoCodecStatus.OK;
    }

    @Override
    public VideoCodecStatus release() {
        return VideoCodecStatus.OK;
    }
}

• The image and its buffer are read-only and valid only until decode() returns; copy the data to keep it longer.
• A decoded frame must carry the capture time of the image it was decoded from as its timestamp.
• onDecodedFrame() takes a reference of its own, so the caller still releases the frame it created. Frame buffers other than NativeI420Buffer are copied, and have to provide direct byte buffers.

Errors

Whatever an encoder, decoder or factory method throws is logged and treated as VideoCodecStatus.ERROR, and a factory that throws from createEncoder() or createDecoder() creates no codec for that stream. On an error other than FALLBACK_SOFTWARE and UNINITIALIZED, WebRTC releases the codec and initializes it again.

Related API

• PeerConnectionFactory.Builder — sets the video encoder and decoder factories.
• VideoEncoderFactory, VideoDecoderFactory — create the codecs of a factory.
• VideoEncoder, VideoDecoder, EncodedImage — codecs implemented in Java.
• DefaultVideoEncoderFactory, DefaultVideoDecoderFactory — the built-in codecs.
• HardwareVideoEncoderFactory, HardwareVideoDecoderFactory — the built-in codecs, on the GPU where possible.
• RTCRtpTransceiver.setCodecPreferences() — chooses among the negotiated codecs.

For the full API, see the JavaDoc of the dev.onvoid.webrtc.media.video.codec package.