Vector Quantisation: How Code Books Compress Similar Data
Vector quantisation helps when many groups of values look alike. An image may contain similar patches of pixels, a signal may repeat familiar shapes, and a collection of feature vectors may cluster around a few common patterns. As a result, storing every group in full can be wasteful when an approximate representation is acceptable.
In practice, vector quantisation replaces each input vector with the index of a representative vector from a shared code book. The index is the compact part. The code book is the shared dictionary that lets a decoder turn that index back into an approximate vector.
Key takeaways
- A code book stores representative vectors called code-vectors.
- An encoder chooses the code-vector that is closest under a chosen distortion measure.
- A decoder uses the transmitted or stored index to look up the selected code-vector.
- Code-book size and quality affect reconstruction detail, storage needs, and encoder work.
How vector quantisation encodes and decodes data
Vector quantisation works on groups of values rather than one value at a time. For example, a two-value vector might represent two related measurements. A longer vector might represent a small block of samples. Before encoding, the system has a code book containing candidate vectors chosen to represent the data.
For each input, the encoder compares that vector with the code-book entries. It then selects the entry with the lowest value under its chosen distortion measure and emits that entry’s index. The decoder needs the same code book. Given the index, it performs a lookup and reconstructs the selected representative vector.
This is the basic model described in the source overview. Likewise, SciPy’s API reference describes assigning codes by comparing observations with code-book centroids.
A vector quantisation example
Imagine a code book with three two-dimensional entries:
0: (1.0, 1.0)
1: (5.0, 5.0)
2: (9.0, 9.0)If the input is (5.2, 4.8), the encoder might decide that entry 1, (5.0, 5.0), is the closest match. Instead of storing the two original numbers, it stores the index 1. The decoder then looks up index 1 and reconstructs (5.0, 5.0).
The numbers here are only an illustration. Even so, they show the important compromise: the reconstructed vector is close to the input, but it is not identical. Therefore, acceptable loss must be defined by the use case and the chosen distortion measure.
How a code book is created
A code book should reflect the kinds of vectors that the system expects to see. One common approach is to run k-means on a set of observation vectors and use the resulting centroids as code-book entries. SciPy documents routines for k-means clustering, generating code books from k-means models, and quantising observations against code-book centroids.
It helps to separate this design stage from day-to-day encoding. First, choose training data, select a code-book size, and construct representative entries. Next, compare each new vector with those entries and assign an index. Consequently, a poor code book can produce poor reconstructions even when the lookup process itself is correct.
Distortion is the price of compact representation
Vector quantisation is generally lossy because it replaces an input with a representative rather than preserving every original value. The distortion measure defines what “closest” means. For instance, a simple choice may measure squared distance between vectors. However, the appropriate measure depends on which differences matter for the data and the reader’s application.
The central decision is a rate-distortion trade-off. A smaller code book can make indices and stored state more compact, but it offers fewer representatives. In contrast, a larger code book can provide a closer match for more inputs. It also requires more code-book storage and may require more work to search for the nearest entry. The source overview notes that code-book construction and nearest-code-vector search can be computationally demanding for the encoder, while decoding is a table lookup.
When vector quantisation is a useful fit
Vector quantisation is worth considering when approximate reconstruction is acceptable and the data contains recurring patterns that a shared code book can represent. Examples can include blocks of image pixels, groups of speech samples, and feature vectors. However, the method does not promise a particular compression ratio or quality level. Those outcomes depend on the data, the code book, and the distortion measure.
It is less suitable when every original value must be recovered exactly or when no sensible definition of acceptable error is available. In those cases, a lossless representation or a different compression strategy may be the better starting point.
Start with a small measurable experiment
Vector quantisation is best understood as a controlled approximation. Use it if you are evaluating compact representations of repeated patterns and can measure the loss that matters to your application. Otherwise, do not use it where exact reconstruction is non-negotiable.
The safest first action is to assemble a small representative data set. Then choose a distortion measure, create several code books of different sizes, and compare the reconstruction error. Finally, use that experiment to make the trade-off visible before you commit to a larger implementation.
Categories: Machine Learning
Tags: Artificial Intelligence