Media Files

This guide explains how to send a media file over a peer connection instead of a camera and a microphone, using the webrtc-java-media module. It covers:

• Adding the media module to a project
• Sending a file with MediaFileSource
• Reading what a source contains with MediaReader and MediaInfo
• Controlling playback and following it with a listener
• Feeding your own media sources with MediaPlayer
• Playing a live stream from an IP camera or media server over RTSP

Sending a file is a common need: a test pattern instead of a webcam, a pre-recorded briefing, a video that has to reach several participants. Without help, an application has to bring its own decoder and push I420 frames into a CustomVideoSource itself. The media module removes that work by decoding with FFmpeg inside the library.

Decoding happens entirely in native code. Frames never travel through Java: the module hands decoded pictures straight to the native side of a CustomVideoSource, and 10 ms chunks to a CustomAudioSource. They are paced in real time and carry the presentation times of the file, so what a receiver gets keeps the timing of the media rather than the timing of a Java thread.

Adding the Module

The module is part of the normal build, and it builds FFmpeg and dav1d (the software decoder of AV1) from submodules pinned to release tags, so the submodules have to be present:

git submodule update --init --depth 1 webrtc-java-media/third-party/ffmpeg webrtc-java-media/third-party/dav1d
mvn install

Building FFmpeg needs make and nasm, and building dav1d needs Meson (pip install meson) and Ninja. On Windows make and nasm come from MSYS2:

winget install MSYS2.MSYS2
C:\msys64\usr\bin\bash -lc "pacman -S --needed make nasm diffutils pkgconf"

Maven still runs from an ordinary shell; the build enters MSYS2 and the Visual Studio environment on its own. The first build compiles FFmpeg and dav1d, which takes a while; later builds reuse the install directory.

Once installed, depend on it alongside webrtc-java. It takes two entries: one for the Java API, and one for the natives of the platform you are running on.

<dependency>
    <groupId>dev.onvoid.webrtc</groupId>
    <artifactId>webrtc-java-media</artifactId>
    <version>0.20.0</version>
</dependency>
<dependency>
    <groupId>dev.onvoid.webrtc</groupId>
    <artifactId>webrtc-java-media</artifactId>
    <version>0.20.0</version>
    <classifier>windows-x86_64</classifier>
</dependency>

The classifier jar carries the module's native library together with the FFmpeg shared libraries it uses, so applications that do not use this module never download FFmpeg. Replace the classifier with the platform you are building for: windows-x86_64, windows-aarch64, linux-x86_64, linux-aarch64, linux-aarch32, macos-x86_64 or macos-aarch64.

INFO

Unlike webrtc-java, which brings its natives along by itself, this module cannot: the natives are built by the module rather than by a separate one, so a dependency on them would have nothing to resolve against on a first build. Asking for them explicitly is the price of that.

Sending a File

MediaFileSource is the short way to do all of it. It opens the source, creates a media source for each kind of media the file actually has, and wires a player to feed them:

// Import required classes
import dev.onvoid.webrtc.media.player.MediaFileSource;
import dev.onvoid.webrtc.media.audio.AudioTrack;
import dev.onvoid.webrtc.media.video.VideoTrack;
import java.nio.file.Path;
import java.util.List;

MediaFileSource source = new MediaFileSource(Path.of("movie.mp4"));

// Create tracks from the media sources the file feeds.
VideoTrack videoTrack = factory.createVideoTrack("video", source.getVideoSource());
AudioTrack audioTrack = factory.createAudioTrack("audio", source.getAudioSource());

peerConnection.addTrack(videoTrack, List.of("stream"));
peerConnection.addTrack(audioTrack, List.of("stream"));

// Start sending.
source.play();

A file with no audio has no audio source, and likewise for video, so check before making a track:

if (source.getVideoSource() != null) {
    VideoTrack videoTrack = factory.createVideoTrack("video", source.getVideoSource());
    peerConnection.addTrack(videoTrack, List.of("stream"));
}

A factory sends either pushed audio or captured audio

A PeerConnectionFactory fed from this module is sending pushed audio and cannot also send audio captured by its AudioDeviceModule. Give such a factory a dummy audio layer, and use a second factory if an application needs both:

AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio);
PeerConnectionFactory factory = new PeerConnectionFactory(audioModule);

Reading What a Source Contains

MediaReader opens a source and reports what is in it, without playing anything:

// Import required classes
import dev.onvoid.webrtc.media.player.MediaInfo;
import dev.onvoid.webrtc.media.player.MediaReader;

try (MediaReader reader = new MediaReader(Path.of("movie.mp4"))) {
    MediaInfo info = reader.getInfo();

    System.out.println("Runs " + info.getDurationUs() / 1_000_000.0 + " s");

    if (info.hasVideo()) {
        System.out.println(info.getVideoWidth() + "x" + info.getVideoHeight()
                + " at " + info.getFrameRate() + " fps, " + info.getVideoCodec());
    }
    if (info.hasAudio()) {
        System.out.println(info.getSampleRate() + " Hz, "
                + info.getChannels() + " channels, " + info.getAudioCodec());
    }
}

A MediaFileSource already has this, so there is no need to open the file twice:

MediaInfo info = source.getInfo();

INFO

Duration is 0 for a source whose container does not say how long it runs, which is the case for live streams. Opening fails if a source holds nothing that can be played.

Controlling Playback

source.play();                  // start, or resume after a pause
source.pause();                 // hold playback where it is
source.seek(30_000_000);        // move to 30 seconds
source.setLooping(true);        // start over instead of ending
source.getPositionUs();         // where playback has got to
source.getState();              // IDLE, PLAYING, PAUSED, ENDED or CLOSED

Looping keeps the timing across the seam, so a looping file works as a stand-in for a camera that never stops. A looping source never reports an end of stream.

A seek lands on the keyframe at or before the position asked for, which is how far back the decoder has to go to produce a picture at all. How close that is to the position asked for depends on how often the media was encoded with keyframes.

Following Playback

// Import required classes
import dev.onvoid.webrtc.media.player.MediaPlayerListener;
import dev.onvoid.webrtc.media.player.MediaPlayerState;

source.setListener(new MediaPlayerListener() {

    @Override
    public void onStateChanged(MediaPlayerState state) {
        System.out.println("Player state: " + state);
    }

    @Override
    public void onEndOfStream() {
        System.out.println("The source ran out.");
    }

    @Override
    public void onError(String message) {
        System.out.println("Playback failed: " + message);
    }
});

WARNING

Every call arrives on the player's own thread, and that thread is the one decoding the media. A listener must return promptly, and must not wait on the player.

Closing the player, or the MediaFileSource, from a listener is fine: it takes effect at once, and the native player is released on another thread once the listener has returned.

Feeding Your Own Media Sources

MediaFileSource creates the media sources for you. An application that needs its own — to feed a source it already created, or to send only the video of a file — uses MediaPlayer directly:

// Import required classes
import dev.onvoid.webrtc.media.player.MediaPlayer;
import dev.onvoid.webrtc.media.player.MediaReader;
import dev.onvoid.webrtc.media.video.CustomVideoSource;

CustomVideoSource videoSource = new CustomVideoSource();

// Video only: passing null for the audio source decodes and drops the audio.
MediaPlayer player = new MediaPlayer(new MediaReader(path), videoSource, null);

VideoTrack videoTrack = factory.createVideoTrack("video", videoSource);

player.play();

INFO

The player takes over the reader it is given. That reader must not be used or closed afterwards; closing the player releases it.

It also keeps the native side of the media sources it feeds alive until it is closed, so disposing of a source or its track while the player runs is safe.

Decoding in Hardware

Decoding H.264, H.265/HEVC, VP9 and AV1 takes a good part of a processor at high resolutions, and for every stream of a camera wall. A player can decode them on the media engine or GPU of the machine instead. It does so when asked to, with a flag on MediaFileSource or MediaPlayer:

// A file whose video is decoded in hardware, where the platform can.
MediaFileSource source = new MediaFileSource(path, true);

// The same for a player of your own.
MediaPlayer player = new MediaPlayer(new MediaReader(path), videoSource, null, true);

// What is actually in use: false if the platform or the stream has no
// hardware decoder for it.
boolean hardware = source.getPlayer().isHardwareDecoding();

The pictures are the same ones software decoding gives; in the tests, every frame of media in each of these codecs is compared with its software counterpart (for HEVC, with checksums of the pictures FFmpeg's software decoder gave once). Hardware decoding is off unless asked for, except for H.265/HEVC.

HEVC is decoded in hardware only. A player decodes it on the media engine or GPU whether or not the flag is set, and never in software: isHardwareDecoding() is always true for it. Where the machine has no hardware decoder for HEVC, creating the player fails with an IOException; where the hardware does not take a stream (a profile it cannot decode) or fails while playing, playback stops with an error. The fallback below does not apply to it.

• Platforms: macOS through VideoToolbox, Windows through Direct3D 12 (Windows 10 version 2004 or later, with a driver that decodes video through it) and otherwise Direct3D 11, and Linux (x86-64 and ARM64) through NVDEC on NVIDIA GPUs (the driver provides libcuda.so.1 and libnvcuvid.so.1) and through Vulkan Video on the GPUs of Intel and AMD (the Vulkan loader libvulkan.so.1 and a driver with the video extensions of the codec: Mesa's ANV or RADV, recent enough for each codec, AV1 and VP9 needing the newest). These libraries are loaded when a player asks for hardware, so a machine without them is not affected. The flag is accepted everywhere; where the platform has no hardware decoder, or the FFmpeg build has none for it (32-bit ARM), the video is decoded in software, except HEVC, and isHardwareDecoding() says so.
• Codecs: H.264, H.265/HEVC, VP9 and AV1, where the GPU decodes them: a GPU that is a few years old may lack AV1, and on a Mac only an M3 or later has it. VP8, MPEG-4 and MJPEG are decoded in software, and so is AV1 where there is no GPU decoder for it (it is decoded by dav1d, which is built into the FFmpeg libraries).
• Fallback (every codec but HEVC): a stream the hardware does not take, such as a profile it cannot decode, is decoded in software without the player noticing more than isHardwareDecoding() turning false. If the hardware fails before its first picture, the packets it was sent are decoded again in software and none is lost; if it fails after one, decoding goes on in software from the next key frame. A damaged packet is not taken for a hardware failure after the first picture: as with software decoding, playback stops with an error, and the hardware keeps the stream.
• What it saves is processor time, not the copy: a decoded picture is read back from the media engine into system memory, and converted to I420 as WebRTC wants it. On an Apple M2, a 1080p H.264 stream took 11.6 ms of processor time per frame in software and 1.4 ms in hardware, and a 4K stream 35.7 ms against 4.2 ms, about seven to eight times less. Per frame, software on all cores is faster on the clock, which does not matter at playback speed. Windows gains less: on an NVIDIA GeForce RTX 3080 through Direct3D 12, a 1080p H.264 stream took about 2.1 ms of processor time per frame against 3.6 ms in software, and a 4K stream 5.4 ms against 6.6 ms. Direct3D 11 reads the picture back more slowly and took more than software at 4K (12.4 ms), which is why Direct3D 12 comes first. These are figures for one machine each and for synthetic media, not a promise; try it on your own streams before relying on it.

Playing a Live Stream

A source can also be a live stream behind an rtsp:// URL, which is what IP cameras, video recorders and most media servers offer. Everything above works the same way; only the URL differs:

MediaFileSource source = new MediaFileSource("rtsp://camera.local:554/stream1");

Credentials go into the URL, as rtsp://user:password@camera.local/stream1. FFmpeg first asks the server for RTP over UDP, and falls back to RTP interleaved on the RTSP connection over TCP if the server refuses UDP or no UDP packets arrive, which is what gets a stream through most firewalls.

A network source can stall or disappear, so every operation that waits on a source has a time limit: opening it, and then each read during playback. It is 10 seconds unless you pass another:

// Import required classes
import java.time.Duration;

MediaFileSource source = new MediaFileSource("rtsp://camera.local/stream1", Duration.ofSeconds(5));

An unreachable server makes the constructor throw an IOException once that time is up. A stream that stops sending during playback is reported to the listener's onError, and the player is left PAUSED. Closing a player never waits for the time limit: it breaks off whatever the player is waiting for at once.

A live stream has no end and no length: its MediaInfo reports a duration of 0, looping does not apply, and seeking is reported as an error. Pausing stops reading from the server, but the server keeps sending; resuming carries on from where playback paused, which leaves it behind real time by as long as it was paused. To get back to live, close the source and open it again.

Closing

Closing a MediaFileSource stops playback and releases the player along with both media sources. Release the senders a peer connection handed out, and the tracks, before that:

videoSender.dispose();
audioSender.dispose();
peerConnection.close();

source.close();

What Can Be Played

The FFmpeg build is deliberately small, and carries only what this module plays:

Containers	MP4 and MOV, Matroska and WebM, AVI, MPEG-TS, FLV, WAV, MP3, Ogg, FLAC, AAC
Video	H.264, H.265/HEVC (hardware only), VP8, VP9, AV1, MPEG-4 (including Xvid and DivX), Microsoft MPEG-4 v1 to v3, MJPEG
Audio	AAC, MP3, MP2, AC-3, Opus, Vorbis, FLAC, PCM, MS and IMA ADPCM

Audio of any rate or layout is resampled to what WebRTC takes, which is 48 kHz 16-bit PCM in mono or stereo. Video that decodes to I420 — almost all 8-bit H.264, VP8, VP9 and MPEG-4 — reaches the encoder without being copied; anything else is converted first.

Sources can be local files, or live streams over RTSP (rtsp://), with RTP over UDP or TCP. Protocols that need TLS, such as rtsps:// and https://, are not part of the build, and neither are HLS, DASH and RTMP. A source cannot reach any other protocol either, even one FFmpeg uses internally, so a URL passed on from a user cannot make the module fetch something else.

Licensing

The module uses FFmpeg under the LGPL version 2.1 or later. It is configured without --enable-gpl and without --enable-nonfree, and FFmpeg is linked dynamically and shipped as separate files inside the platform jar, so its libraries may be replaced with your own build, as the LGPL requires. The wrapper code is licensed under the Apache License 2.0 like the rest of webrtc-java.

Each platform jar carries the LGPL text and a notice under META-INF/licenses/ffmpeg, naming the FFmpeg release the libraries are built from, unmodified, and where its source is. If you redistribute your application with these jars, keep those files with them.

Some of the formats FFmpeg decodes, such as H.264, H.265/HEVC and AAC, may be covered by patents in some countries. Whether your use of them needs a patent license is for you to determine.

Complete Example

See MediaFileExample in the webrtc-examples module, which opens a file, reports what it contains, creates tracks, adds them to a peer connection and follows playback to the end.

mvn -pl webrtc-examples compile