Home Page | About Me | Home Entertainment | Home Entertainment Blog | Politics | Australian Libertarian Society Blog | Disclosures

Know It All: Data Compression -- Squeezing stuff to size

Published in Geare magazine, Issue #43, 2007

Digital recording is best known for the great quality it provides and, to a lesser extent, its immunity from generational degradation. Every time you copy a recording -- whether sound, video or even a still picture -- from an analogue medium, it degrades. If you copy from a copy, the damage just keeps on piling up. But digital recordings don't suffer from this, because very single 'bit' can be copied perfectly.

An equally important virtue of digital technology is the ability to compress it in ways that can never be done with analogue signals. This is just as well, because digital technology isn't magic. The amount of data space consumed by a digital signal is rougly comparable to that used by an equivalent quality analogue signal, so digital isn't inherently more compact.

But because digital signals are simply very long strings of numbers (a four minute CD track has 21 million numbers lined up in a row), they are amendable to the application of mathematics. And mathematicians have been doing some pretty amazing things in recent years.

Compression of data is needed to allow you to fit a reasonable number of tracks onto your MP3 player, to fit a two hour movie onto a DVD, to allow photos to load from a webpage in seconds rather than minutes or hours, and so on.

There are two kinds of signal compression in common use these days: lossy and lossless.

Lossy compression generally manages to squeeze the signal down to a tiny proportion of its uncompressed size. The usual 128kb/s bitrate for MP3 is just one eleventh of the bitrate of a CD. A photo I took on my digital camera consumed 2,300kB, thanks to JPEG compression. Uncompressed, it swelled out to 17,600kB. I could further compress it in JPEG format to just 530kB, with only the subtlest differences visible. That's a compression ratio of 33:1. MPEG2 video on DVDs is even more highly compressed.

The techniques of lossy compression are more or less the same: they rely on mathematical algorithms identifying stuff in the signal they we won't see or hear anyway, and then tossing it out. For example, different sections of our ears detect different bands of frequencies. If there are two sounds extremely close to each other within one of those bands, and one of the sounds is significantly louder than the other, then we simply don't hear the quieter one ... even if it comes first!

Lossy compression algorithms have to be used in moderation, though. If you overcompress something, it rapidly becomes disturbing. Most damage to signals caused by analogue systems manifests in the form of either white noise, or harmonic distortion. Both of these are fairly tolerable because they are naturally occurring noises. But use excessive compression in a JPEG picture, or an MP3 music file, and the noises generated are not perceptually or musically related to the original signal. In JPEG files you get random dots gathered around the edges of visible objects. In MP3 you get a kind of unsteady, unsettling, warble around the attack of sounds.

To achieve these large compression rates, lossy compression is required because, as a mathematician would say, there is little redundancy in these signals. That means that there is little repetition.

Lossless compression, though, is likely to become more common in consumer electronics. Until recently, the only even moderately widespread type of lossless compression in general consumer electronics was the Meridian Lossless Packing used on DVD Audio. Unfortunately, that music format is on life support.

Lossless compression, as its name implies, allows the original signal to be reconstructed without any degradation at all. Some have complained that, despite the science behind such formats as MP3, the losses truly are audible.

MLP pulls of its trick by using a predictive algorithm. If the signal has been increasing for the last few dozens of microseconds, it's reasonable to assume that it will for the next ten microseconds (assuming a 96kHz sample rate). So the algorithm, instead of recording the level of the next moment, records the difference of the actual level from what it predicted. This is typically a much small number, and thus needs far less space to record. If there is an unexpectedly big jump, then the algorithm flags this and makes an allowance for it.

This allows MLP to roughly halve the amount of space required for its high quality music.

The gradual demise of DVD Audio has not led to the demise of the compression format, though. MLP has been repackaged and extended as Dolby TrueHD, and accompanied by the similar DTS HD, on Blu-ray and HD DVD.

© 2002-2009, Stephen Dawson