Trellis quantization is an algorithm that can improve data compression in DCT-based encoding methods.
It is used to optimize residual DCT coefficients after motion estimation in lossy video compression encoders such as Xvid and x264.
Trellis quantization reduces the size of some DCT coefficients while recovering others to take their place.
This process can increase quality because coefficients chosen by Trellis have the lowest rate-distortion ratio.
It has varying effectiveness depending on the input data and compression method.