What is a codec?

Codecs are a core technical consideration for every step of post-production, so it is critical that everyone who touches your workflow understands the basics.

A codec is the set of rules that tells computers and electronic equipment how to handle your media files, most notably digital video footage. The term codec is a shortening of the words compressor-decompressor or coder-decoder. As the name implies, codecs make video files smaller for storage, and then turn the compressed data back into a usable image when you need to use it again.

[Note: Codecs are not the same as containers. Containers are the file types that actually store digital video data, and can be used in conjunction with many different codecs. Imagine containers as the box where you put data, and codecs as the instructions for packing and unpacking the box. H.264, DNxHD, and ProRes are codecs, while .movs, .mxfs are containers.]

We need codecs because uncompressed video files are gigantic. A single minute of uncompressed 4K footage measures in the dozens of gigabytes. So, you will probably never have the option of working with uncompressed footage (video without a codec) all the way through your workflow. It’s just too large and too complex to work with in most cases.

Codecs solve this problem by reducing the size of your footage and making it easier to work with across the post-production pipeline. But just because the image is compressed doesn’t mean it will look worse. While not all codecs are created equal, there are many high-end codecs you can choose from that still deliver stunning image quality. The right codec will make your footage much easier to manage, and you may not even be able to tell a difference from the original.

That said, you will be able to tell a difference if you choose the wrong codec for certain post-production processes. Using a codec with too much compression might make your desired color-correction unachievable, or make your VFX work look unrealistic. On the other hand, codecs with too little compression might make your video uneditable on normal computer hardware, and cause issues when transferring data between teams or facilities. Different codecs are good in different contexts, but there is no single codec that works for every use case.

Choosing codecs is one of the most important technical decisions you will make for your project. To a large extent, codecs determine what you can do with your footage in post and how complex your workflow will be. It is usually much safer, faster, and cheaper to pick codecs that you know will fit neatly into a pre-planned/pre-existing workflow, than it is to improvise a new workflow around codecs that you have not used before. The last thing you want is to find out some or all of your footage is unusable because the codec isn’t compatible with your software or hardware. As with everything in post-production, test all your codec choices before the project begins.

Of course, to pick codecs effectively, you need to understand how they impact the visual qualities of video. While the mathematics and computer science behind codecs are quite complicated, there are only a few fundamental concepts you need to know to choose the right codecs for your production.

Bit Depth

First, you need to understand bit depth (also called color depth). This is basically the number of colors that make up an image. The most common bit depths for video are 8-bit (16 million colors), 10-bit (1 billion colors), and 12-bit (68 billion colors). Higher bit depth codecs can store more colors, which makes video appear smoother and subtler. In general, codecs with a deeper bit depth provide a higher quality image, but they also make for larger files. It is best to use a deep bit depth codec for video capture, color correction, and VFX.

Chroma Subsampling

Next, you need to understand chroma subsampling. This is a method of reducing an image’s size by throwing away the color data of some pixels and then replacing it with adjacent color. Chroma subsampling is expressed as a ratio, like 4:4:4 (no pixels lose color data), 4:2:2 (50% of pixels lose color data), and 4:2:0 (75% of pixels lose color data). This can save a tremendous amount of space, and in many situations will not be noticeable to viewers because the human eye is much more sensitive to brightness than to color changes.

However, you need to be aware of how much different codecs subsample color. Too much can visibly degrade image quality, and will make it difficult to manipulate the image in certain ways. In general, you should pick a codec with as little chroma subsampling as possible for image capture. That will give your original camera files lots of color data, which will make color correction and VFX work much easier. If you shoot a greenscreen shot in a 4:2:0 codec, for instance, you will be in a lot of trouble.

Codec Compression Techniques

Codecs use a variety of advanced computational techniques to compress video data, but there are two primary compression techniques that are important to understand; spatial compression and temporal compression.

Spatial compression is a technique for saving space within individual frames of video. This method is often called intraframe compression, and it works by grouping pixels together and storing them as a single color value, called a block. Blocks take up much less digital storage space than all the pixels would individually, especially when large portions of the image are a similar color already. However, there will be some inherent quality loss for complex images with lots of detail. To overcome this issue, many codecs employ flexible blocks of almost any shape and size, which results in less perceptible loss of quality.

Temporal compression, or interframe compression, is a technique for saving space across multiple video frames. It uses the same block compression techniques as spatial compression, but not for every frame. Instead, temporal compression only saves the difference between sequential frames. If half of an image is the same between two frames, then this technique only saves the new pixels in the second frame, and just fills in the rest with the unchanged pixels of the first frame. Codecs that use this method are called long-GOP (group of pictures) codecs, and they are very efficient at storing information, because they don’t need to store the entirety of every frame. However, because every frame does not contain a complete image, the computational cost is higher if you are working with the footage non-sequentially, like when you are editing. In general, codecs that use temporal compression are great for storing information in a small amount of space, but not so great for working with the footage.

Bit Rate

Finally, you need to understand bit rate. Bit rate measures how much data a codec uses over time, usually expressed as megabits per second (Mbps or Mb/s). A codec’s bit rate determines the visual quality of your video more than almost any other factor–more than resolution, more than your favorite camera and lens combo, and more than your meticulous color work. Why? Because a codec can only use as much data as the bit rate allows it to use. If a codec has a bit rate of 10 Mbps, then 2 minutes of footage can only use 1200Mb of storage (10Mbps x 120 seconds), period. It doesn’t matter if your shooting 720p or 8K, the file size stays the same. Since 8K footage has 36 times more pixels than 720p, the compression will have to be at least 36 times more severe if you try to fit it in the same bit rate as 720p. Yeah, the 8K file might technically still be higher resolution, but it will look awful. When choosing codecs, it’s critical that you test the bit rate options to make sure they can reproduce the level of visual quality you need for your project.

[Note: A megabit (Mb) is not the same as a megabyte (MB). 1 byte is made up of 8 bits. In the previous example, 1200Mb equals 150MB (1200Mb / 8bits = 150MB). The 1:8 conversion of bytes to bits is the same regardless of the binary prefix.]

Types of Codecs

As we established already, there is no single codec that is good for every part of the workflow. Simply put, the codec you use in-camera will not be the same codec that makes it to a viewer’s screen. The characteristics and specifications you need for these two applications (and every step of post-production in between) are radically different. If you tried to use the same codec for every step of post-production, you would have to sacrifice something along the way. So, you will need to pick codecs that make each part of your workflow the easiest, while also maintaining quality.

Your codec journey begins in the camera, with the codec you use to capture your footage. The best capture codecs have deep bit depths, low chroma subsampling, high bit rates, and do not use long-GOP compression. There is a huge variety of capture codecs, but if it meets these criteria then it’s likely a viable option. As a rule, you will want to use the best codec you can for capture, because it will be the source for all other codecs you transcode to down the line. You can always convert high quality footage to lower quality files, but it doesn’t work the other way around.

Once your footage is offloaded from your camera, your next codec choice will be the codec you use to edit your footage. Editing requires speed and ease-of-use above image quality, so it’s a good idea to transcode your original camera files to a codec that generates much small files. However, be sure to avoid long-GOP codecs (like most versions of H.264) for editing. These codecs do make small files, but your computer will have to work much harder to keep up with all the referenced information as you scrub around the timeline. That may slow down your editing significantly, so make sure your editing codec only uses spatial compression (like DNxHD or ProRes). Also, don’t be embarrassed if your computer needs to user a lower-quality codec for editing. It was only a few years ago that even big-budget Hollywood movies were being edited at 480p. It’s just that much faster and easier. Besides, once you’ve locked down the edit, you will be relinking to the higher quality files anyway that will replace the low-quality editing codec.

After the edit is complete, you will need a codec for color correction and VFX. Often times you can just use the original camera files that were conformed into your video timeline, which ensures you are getting the best quality image possible. But if you decide to use a separate codec for these processes, you will want one that is preserves even more information than the capture codec.

When the time comes to export your video for delivery, you might think that a bigger codec is always better. In reality, though, the delivery medium always dictates the codec. We cover those concepts in more detail in the delivery section.

Once you deliver the final version of your video to its intended destination(s), you will need to export in a different codec for archiving. As happens in this industry, the software and hardware we use changes frequently, and clients sometimes ask for a higher resolution or higher quality files years after delivery. In those cases, opening the original project files and re-exporting is not always possible. So, you need a good codec for your archived files. It should be extremely high quality, with as high a bit rate as possible, so that you will get the best possible results when you need to transcode it in the future. It’s usually worth the tradeoff of storing a very large file.

There are hundreds of codecs out there, and at times it can seem overwhelming. You might wonder how you’re even supposed to find the best options for your project. However, there is another factor to consider when picking codecs that should make choosing them easier; their popularity. Popular codecs are more likely to work on your computer, on your client’s computer, and on both of your new/upgraded computers five years from now. Also, popularity breeds support. If something goes wrong or you have a question, it’s easier to find help if thousands of other users are posting about it online. When you’re picking codecs for every step of your project, you can save a lot of time (and heartache) by using codecs that are popular.

If you are ready to pick codecs for your project, you should check out the Frame.io Codec Comparison. This tool will give you a good idea of how the most popular codecs handle your footage, and help you decide what works best for your workflow.