The State of Video Codecs in 2024

Introduction

Internet speed and accessibility are major catalysts behind the current success of video streaming. However, what many consumers take for granted is the encoding and decoding technology that makes modern video streaming possible. As with most technology, this is an evolving field. To understand the best practices for developing and packaging digital video content, it’s important to understand the current state of video encoding, decoding, and transcoding, along with the codecs that make them possible.

What is Video Codec?

A video codec is a software or hardware component that is used to encode and decode digital video data. It performs the process of compressing and decompressing the video data, allowing it to be transmitted more efficiently over the internet or stored on a device.

Difference Between Codec and Container

Codec and container are often used interchangeably, but they refer to different things. A codec refers to the specific algorithm that compresses and decompresses the video data. At the same time, a container is a file format that holds the compressed video data along with audio, subtitle, and metadata information.

Here is a table of common video codecs with container support data:

It is important to note that different codecs have different levels of compression efficiency, quality, and hardware support. When choosing a codec, one must consider factors such as compatibility, bandwidth requirements, and desired video quality.

Different Types of Video Codecs

There are three main types of codecs in video production and post-production. These can be software or hardware-based. They are as follows:

• Compression: Compression codecs are the most widely used type among everyday users and are designed to reduce the file size of videos while maintaining acceptable quality. They’re often used to compress media into a specific data storage or streaming format. The format into which they’re compressed must follow a documented standard so that it can be interpreted by other devices or software.
• Raw: Raw codecs are typically used in video capturing and recording. They store the data your camera’s image sensor records without converting or compressing it. This allows you to have full editing control during the post-production process.
• Mezzanine: Mezzanine codecs, also known as editing or intermediate codecs, are intended for post-production. They retain the quality from the source recording but optimize it for storage and playback. While they result in smaller file sizes than raw codecs, they are still larger than compressed delivery formats. They’re also less resource-intensive because they don’t require decoding for playback.

As noted, many codecs cause degradation of fidelity or quality. This degradation is called lossiness. While lossless codecs that don’t cause quality degradation exist, they often result in larger files that take more computing resources to decode or stream.

Why are Video codecs important?

Here are some reasons why video codecs are important:

• Video codecs help reduce the amount of data needed to store or transmit video content, making it more efficient and cost-effective.
• Video codecs provide better compression ratios, allowing higher-quality video to be streamed or stored at lower bitrates.
• Codecs improve video quality by reducing artifacts and other distortions that may occur during encoding and decoding.
• Video codecs also increase compatibility with different devices and software by supporting various video formats and standards.
• Codecs enable the creation of new video formats and applications, such as virtual reality and augmented reality experiences.

Comparing Modern Compression Video Codecs

Lossy-ness and the intended use of the encoded file are just some of the factors you need to consider when selecting the right codec for your workflow. Others include support, compatibility, latency, playback quality, encoding speed, decoding performance, and cost. In this section, you’ll explore some of the most popular modern video codecs.

All encoding tests were performed on the Blue Sky and Clouds Timelapse video. The raw footage is 55.9 MB.

The system specs for the encoding tests were as follows:

• Operating System: Windows 10 Home Edition (19043.1645)
• Processor: Intel® Core™ i7-8550U CPU @ 1.80GHz 1.99 GHz
• Installed RAM: 8 GB
• Graphics: Intel® UHD Graphics 620

H.264 Codec

H.264 was first released in August 2004, and version 14.0, the most recent version as of this writing, was released in August 2021. Despite the release of newer and supposedly more efficient codecs, the H.264 standard, also known as Advanced Video Coding (AVC) or MPEG-4 Part 10, still has the largest market share when it comes to online videos. As such, it has near-universal device support.

It provides ample compression and decent playback quality. This makes it a safe bet for neophytes unsure of which codec to use for their project. Furthermore, if you’re capturing video on a modern phone or camera, you’re likely using H.264. Because of its ubiquity and sustained popularity, it does not look like H.264 standard will soon be deprecated.

Platform Support

The H.264 codec is broadly supported. All major desktop and mobile operating systems support it, as do most popular content delivery networks and web browsers.

Implementations

Common software implementations of the H.264 codec include x264, Nero Digital, MainConcept, OpenH264, and DivX. Hardware implementations include Intel Quick Sync, AMD Video Coding Engine (VCE), Nvidia NVENC, and ARM (Media Foundation).

System Requirements

Software encoding through the CPU requires a system with at least the following specifications:

• A dual-core CPU with at least a 2.0 GHz clock speed, such as the Intel Core 2 Duo or AMD Athlon X2
• 1 GB RAM for 1080p 30 FPS Video
• 2 GB RAM for 4K 30 FPS Videos

Usage Costs

Licensing for AVC/H.264 is handled by MPEG-LA. As of 2016, H.264’s royalty structures were as follows:

• Per-Device Royalties: $0.10–$0.20 USD per unit, depending on volume. There are no royalties charged on volumes less than a hundred thousand units.
• Per-Title Royalties: The lesser of $0.02 USD or 2% of the title sale value. There is no yearly cap on this figure.
• Subscription-Based Royalties: $25,000–$100,000 USD per year, depending on the number of subscribers. No royalties are charged to organizations with fewer than a hundred thousand subscribers.
• -Free/Public Over-the-Air Broadcast: One-time fee of $2,500 USD per encoder, or $2,500–$10,000 USD per year, based on the number of households in the broadcast area.

As of 2016, there is a yearly cap of $9,500,000 USD for all royalties.

Encoding Speed

The following figures were obtained using the Handbrake Video Transcoder using the following settings:

• Software video encoder: H.264 (x264)
• Frame rate per second (FPS): 30 (peak frame rate)
• Avg Bitrate: 4500 kbps
• 2-Pass encoding: On
• -Turbo first pass: On
• Encoder Tune: None
• Encoder Level: Auto
• Peak Framerate: On
• Scaled Size: 1920 x 1080
• Format: MP4
• Audio Track: None
• Subtitle Tracks: None
• Web Optimized: On
• Align A/V Start: On
• iPod 5G Support: On
• Passthru Common Metadata: On
• Filters: Decomb

The results of the test were as follows:

• Average encoding speed: 17.502253 fps
• Frame output: 390 frames
• Total elapsed time: Fifty-six seconds
• Final file size: 8.40 MB
• VMAF Score: 92.256554

Note: Since this test uses x264 (software encoding), the encoding process exclusively utilizes the CPU’s capabilities. VMAF scores were obtained using FFmpeg’s libvamf (vmaf_v0.6.1).

H.265 Codec

H.265, also known as HEVC, is a successor to H.264 and was released in June 2013. Despite being nearly a decade old, it is still considered a new codec due to its slow adoption and lack of device support, making it more demanding on computing resources. To make it difficult to work with H.265 encoded files on older systems, you must transcode the video first.

Another reason for the slow adoption is that systems running older CPUs may struggle because they aren’t optimized for H.265. Intel introduced hardware-accelerated support with its sixth-generation Intel processors in 2015, while Apple's M1 chips can easily encode H. 265 4K videos. Some professional streamers, video editors, and some users running older machines have managed this limitation with the support of a hardware encoder, which allows for real-time publishing and encoding of streams in real-time, freeing up system resources for content production.

H.265 provides an additional 20-50% reduction in bit rate while retaining the same picture quality, resulting in lower bandwidth requirements for video streaming. For example, HD video encoded with H.264 would require six Mbps, while H.265 would only require three Mbps. In a more perceptible example, an H.264 encoded 4K video would require thirty-two Mbps to stream, while its H.265 counterpart would only require fifteen Mbps.

Platform Support

H.265 is supported by most modern operating systems. It’s supported by most distributions of Linux, macOS High Sierra or later, Windows XP or later, iOS 11 or later, and Android 5.0 or later. Some content delivery networks and most modern web browsers also support it.

Implementations

The primary software implementation of H.265 is x265. Hardware implementations include Intel Quick Sync, AMD Video Coding Engine (VCE), Nvidia NVENC, and ARM (Media Foundation).

System Requirements

Minimum supported processors and hardware:

• Nvidia GM206 Graphics Processing Units
• 6th generation Intel Processors
• 6th generation AMD Accelerated Processing Units
• AMD Fiji GPUS
• Nvidia Tegra X1 System-on-Chip packages
• Qualcomm Snapdragon 208/410/615/805 System-on-Chip packages
• Samsung Exynos 5 Octa 5430 System-on-Chip packages
• Apple A8 System-on-Chip packages
• Some MediaTek system packages that were produced post-2015

Minimum RAM:

• 2 GB for 1080p 30fps videos
• 4 GB for 4K 30fps videos

Usage Costs

HEVC/H.265 patents are licensed by two main organizations: MPEG-LA and Access Advance, formerly known as HEVC Advance. As of 2016, H.265’s royalty and licensing structures were as follows:

• Per-Device Royalties: $0.20 USD per unit for volumes of less than a hundred thousand units for MPEG-LA patents, with a cap of $25,000,000 USD a year. $0.40–$1.20 USD per unit, depending on the volume, for Access Advance patents. Access Advance also charges an additional $0.10–$0.75 USD for higher HEVC profiles. Access Advance has a cap of $20,000,000–$30,000,000 USD for mobile devices and $20,000,000 USD for other devices. MPEG-LA has no other royalty fees.
• Per-Title Royalties: Access Advance charges $0.25 USD per unit, with a yearly cap of $2,500,000 USD.
• -Subscription-Based: As of 2020, Access Advance charges $0.025 USD per subscriber. There is a yearly cap of $2,500,000 USD.

Access Advance has no broadcast fees and has a per-organization yearly royalty cap of $5,000,000 USD for titles and subscriptions and $40,000,000 USD for devices.

Encoding Speed

The following results were obtained using H.265 (x265) as a software encoder. The settings and specifications were otherwise carried over from the x264 test.

The results of the test were as follows:

• Average encoding speed: 10.216110 fps
• Frame output: 390 frames
• Total elapsed time: One minute, forty-three seconds
• Final file size: 8.30 MB
• VMAF score: 94.476300

The encoding took longer than it did with x264, but the resulting file was, as expected, smaller. Although it isn’t significantly smaller, it has better video quality according to the VMAF score.

AV1 Codec

AOMedia Video 1 (AV1) was released in March 2018, making it one of the most recent codecs on this list. The Alliance developed it for Open Media (AOM), a non-profit organization whose members include Apple, Facebook, Netflix, Google, and Samsung. Despite its impressive list of backers, adoption and support for AV1 are not as widespread as the other codecs on this list, largely because it’s a relatively young project. AV1 was created to be VP9’s successor and was primarily developed for encoding streams over the internet. Accordingly, it promises a reduction of thirty to forty percent in bandwidth compared to HEVC.

It’s been shown to produce compression rates that are fifty percent higher than x264 and thirty-four percent higher than VP9, placing it in direct competition with x265. In addition to its impressive performance, AV1 may be preferable to HEVC and AVC because of its royalty-free licensing model, which allows for easier adoption. Despite this, open-source transcoding projects such as Handbrake have slowly incorporated it for encoding, although at least Handbrake offers to decode support.

AV1 offers better compression than the older codecs but can be incredibly slow. Its royalty-free licensing model means that streaming services are more likely to adopt in the future than they are H.265 or H.266. Netflix has already started using AV1 on some videos for Android devices. Twitch may also offer AV1 support soon.

Platform Support

AV1 is supported by Windows 10 and later through a downloadable add-on, Linux and BSD, Chrome OS version 70 and Android 10 and later, and the latest Firefox, Chrome, and other versions of Chromium-based browsers. Many content-delivery networks also support it.

Notably, despite Apple’s membership in the AOM, neither iOS nor macOS devices offer support for AV1.

Software Implementations

Software implementations include Libaom, rav1e, SVT-AV1, dav1d (decoding only), Cisco AV1, and libgav1. Hardware implementations include:

• Intel QSV (as of generation 11 CPUs and GPUs).
• AMD VCN (for AMD RDNA GPUs, and limited to decoding only).
• ARM64.
• Nvidia NVENC (as of RTX 30 Series, and again limited to decoding only).

System Requirements

Supported processors and hardware:

• Allegro AL-E195 (encoding only)
• AMD Radeon RX6000 series (excluding Navi 24; limited to decoding)
• Amlogic
• S905X4 (decoding only)
• S908X (decoding only)
• S805X2 (decoding only)
• Amphion CS814 (decoding only)
• Broadcom BCM7218X (decoding only)
• Chips&Media
• WAVE510A (decoding only)
• WAVE627 (encoding only)
• Dwango (encoding only)
• Intel
• Intel Gen 12 CPUs and GPUs (decoding and encoding)
• Arctic Sound-M (encoding only)
• MediaTek Dimensity 1000 and later (decoding only)
• Nvidia Geforce 30 series (decoding only)
• Realtek
• RTD1311 (decoding only)
• RTD2893 (decoding only)
• Rockchip RK3588 (decoding only)
• Samsung Exynos 2200 (decoding only)
• Google Tensor (decoding only)

Usage Costs

Much like the V9 codec, it’s royalty-free. Once again, though, third-party implementations of the V9 standard require license fees. Sisvel is responsible for the royalty structures. Royalties are charged for consumer devices on a per-product basis.

• Consumer display devices: €0.24
• Consumer non-display devices: €0.08

Encoding Speed

The following test was conducted using Moisés Cardona’s Rav1e GUI paired with the latest version of Rav1 and FFMPEG. It uses an average video bitrate of 4500 kbps with the highest speed setting selected (10). Additionally, it uses .webm as a container file.

The results of the test were as follows:

• Average encoding speed: 0.046 fps
• Frame output: 390 frames
• Total elapsed time: Thirty-two minutes, forty-one seconds
• Final file size: 8.33 MB
• VMAF score: 93.930335

As you can tell, the encoding process takes much longer than the other codecs on this list. It produces a smaller file size than the H.264 compression but with a slightly lower VMAF score. However, this difference should be invisible to the average viewer.

VP9 Codec

Released in June 2013, VP9 was originally envisioned as an alternative to the H.264 and H.265 compression standards. Developed by Google, it’s famous for being YouTube’s go-to codec for HD content. It delivers encoding performances that are quite similar to H.265.

It improves upon VP8, its predecessor, in terms of final file size but consumes more power and takes longer to encode. VP9 has significantly more web support than H.265—for instance, it has native HTML5 video browser support. Furthermore, it’s had Android support since version 4.4 (KitKat); this can again be credited to Google’s influence.

Platform Support

VP9 is generally well-supported. It’s supported by Windows 10 (since version 1607), macOS 11.0 and later, Linux, Android 4.4 and later, iOS 14.0 and later, many popular content delivery networks, and most major browsers, including Chrome and other Chromium-based browsers, Firefox, and Safari.

Implementations

Software implementations of VP9 include libvpx, SVT-VP9, and Eve. Hardware implementations include Intel QVC (for encoding and decoding), AMD Video Coding Engine (decoding only), Nvidia NVENC (decoding only), and ARM (decoding only).

System Requirements

Minimum supported processors:

• ARM Mali-V61 (Egil) VPU
• HiSilicon HI3798C
• 7th Generation Intel CPUs
• MediaTek Helio X30
• Helio P30
• Qualcomm Snapdragon 835
• Samsung Exynos 9 Octa 9810

RAM:

• 2 GB for 1080p 30fps video
• 4 GB for 4K 30fps videos

Usage Costs

VP9 is offered royalty-free from Google. However, third-party implementations and patents may be subject to a license fee. Sisvel is the organization responsible for the patent pool. They only charge royalty fees on a per-product basis for consumer devices sold.

• Consumer display devices: €0.24 per device
• Consumer non-display devices: €0.08 per device

Encoding Speed

The following results were obtained using similar settings to the H.264 test. However, the test used an MKV container.

The results of the test were as follows:

• Average encoding speed: 1.646879 fps
• Frame output: 390 frames
• Total elapsed time: Five minutes, eighteen seconds
• Final file size: 8.37 MB
• VMAF Score: 94.905509

VP9 took substantially longer than the H.264 and H.265 tests. However, it produced a slightly smaller file than the H.264 test (but a larger one than the H.265 file). Nevertheless, the output file’s video quality was superior to the previous tests.

What about H.266 Codec and MPEG-5 (EVC)?

VVC has been touted as H.265’s successor. A 4K video that requires 6 GB of data in HEVC will maintain the same quality in only 3 GB with VVC. The biggest disadvantage of VVC is that it requires a paid license, which will likely lead some users to opt for less expensive options to deliver content. However, VCC’s compression will be increasingly important as the implementation of 8K resolution becomes more extensive. Codecs such as H.266 will be crucial for online content delivery through video.

H.264 still has a large piece of the video codec market, and one of the reasons that H.265 adoption has been sluggish is due to disputes regarding licensing. H.266 may be a leap in the right direction in terms of technology, but some commentators believe that it may suffer the same fate as H.265 since it will be utilizing a similar licensing model.

AV1 and H.266 are often considered the future of streaming, but in recent years, MPEG-5 has presented itself as a viable alternative to both. Work on MPEG-5, or Essential Video Coding (EVC), began in 2018 by the Moving Pictures Expert Group. It’s quickly seen some success, with brands such as Samsung, Huawei, and Qualcomm committing to the licensing terms.

As with VVC and AV1, MPEG-5 promises a bandwidth reduction of up to twenty-five percent compared to H.265. Contrary to how most codecs are developed, MPEG-5 applies new techniques to standards established by older codecs such as HEVC.

Despite MPEG’s early popularity, industry experts believe that MPEG-5 will be the least successful of the three primary next-gen codecs, as it doesn’t offer any perceivable advantages over its competition.

Which Video Codec is Right for You?

Which codec you ultimately select depends on your goals. Regardless, there are three factors you need to consider when selecting a codec:

• Playback quality
• File size
• Ease of editing
• Codec support

While the final criteria are easily met, most codecs can only deliver two of the first three attributes. For instance, if you want a codec that provides high playback quality and ease of editing, your best bet is a mezzanine codec, such as Apple ProRes or Avid DNxHD. Files encoded using these codecs require a large amount of storage space, which can be expensive. Still, professionals working on videos in post-production may find it a small price to pay for a more efficient workflow.

Compression codecs such as H.264 or H.265 will be your best option if you want video files that deliver high-quality playback in a smaller package. H.264 still balances speed with performance and has the widest support.

H.265 offers better compression but requires marginally more time and resources to perform encoding. Another drawback is that H.265 isn’t supported by many older systems, making H.264 your safest bet.

A proxy workflow may be the best option for optimizing for ease of editing and small file size. You can use Apple ProRes or Avid DNxHD to transcode 4K videos into 720p proxies, edit the proxy, and then re-link it to the original video, which will apply the edits from the proxy. This workflow is a particularly useful solution for professionals working with remote editors, though the drawback is that the transcoding requires a lot of time.

Some social media platforms and content delivery services transcode or encode your content when you upload it. These services use standards such as VP9 to store your content. If you’re a content creator, your main concern will probably be how you store media before you upload it. Again, your best choice would be H.264 or a legacy codec like Theora.

Applications of Video codecs

Some of the key applications of video codecs include:

• Video Streaming: Codecs like H.264, VP9, and AV1 is used to encode and decode video content for streaming services like Netflix, YouTube, and Hulu. These codecs are optimized for low latency and high compression, allowing video content to be streamed efficiently over the internet with minimal buffering.
• Video Conferencing: Codecs like H.264 and VP8 are used in video conferencing applications like Skype, Zoom, and Google Meet. These codecs allow real-time video communication over the internet, with low latency and high image quality.
• Webcasting: Codecs like H.264 and VP8 are used in webcasting applications to broadcast live events over the internet. These codecs are designed to handle large amounts of data, ensuring that the video stream is transmitted smoothly and with minimal buffering.
• Video Editing and Post-Production: Codecs like ProRes, DNxHD, and CineForm are used in video editing and post-production applications like Final Cut Pro, Adobe Premiere, and DaVinci Resolve. These codecs are optimized for high image quality and efficient data storage, making them ideal for video editing and post-production workflows.

Conclusion

AV1 is a strong contender for the default codec for future encoding and streaming. While it may not be as efficient as VVC, its royalty-free licensing model will be far more enticing to developers and streaming service providers. Furthermore, with remote working and video call software becoming more prevalent, AV1 is the logical choice.

As video and compression standards change and evolve, keeping up can be extremely hard. As such, you may benefit from employing a service like Gumlet to handle the delivery of your video content. In addition to choosing the right codec for your workflow or project, Gumlet will also optimize your video content and provide helpful insights into how it is consumed, helping you take your content to the next level.