Kolmogorov–Zurbenko filter

Standard fast Fourier transform (FFT) was completely fooled by the noisy and non-stationary ocean environment.

Filter construction relied on the main concepts of the continuous Fourier transform and their discrete analogues.

Another nice feature of the KZ filter is that the two parameters have clear interpretation so that it can be easily adopted by specialists in different areas.

I.Zurbenko Postdoctoral position at UC Berkeley with Jerzy Neyman and Elizabeth Scott provided a lot of ideas of applications supported in contacts with Murray Rosenblatt, Robert Shumway, Harald Cramér, David Brillinger, Herbert Robbins, Wilfrid Dixon, Emanuel Parzen.

The impulse response function of the KZ filter has k − 2 continuous derivatives and is asymptotically Gaussian distributed.

For practical purposes a choice of k within a range 3 to 5 is usually sufficient, when regular MA (k = 1) is providing strong spectral leakage of about 5%.

Another nice feature of the KZ filter is that the two parameters each have clear interpretations so that it can be easily adopted by specialists in different areas.

Software implementations for time series, longitudinal and spatial data have been developed in the popular statistical package R, which facilitate the use of the KZ filter and its extensions in different areas.

[1] At the same time, because of its natural form, it has computational advantages, permitting analysis of space/time problems with data that has much as 90% of observations missing, and which represent a messy combination of several different physical phenomena.

[2][3] Unlike some mathematical developments, KZ is adaptable by specialists in different areas because it has a clear physical interpretation behind it.

The KZFT is essentially a band pass filter that belongs to the category of short-time Fourier transform (STFT) with a unique time window.

KZFT readily uncovers small deviations from a constant spectral density of white noise resulting from computer random numbers generator.

The average of the square of KZFT in time over S periods of ρ0 = ⁠1/ν0⁠ will provide an estimate of the square amplitude of the wave at frequency ν0 or KZ periodogram (KZP) based on 2Sρ0 observations around moment t: Transfer function of KZFT is provided in Figure 2 has a very sharp frequency resolution with bandwidth limited by c/(m√k).

It clearly display that it perfectly captured frequency of interest ν0 = 0.4 and provide practically no spectral leakage from a side lobes which control by parameter k of filtration.

For practical considerations, the percentage of missing values was used at p=70% to determine if the spectrum could continue to capture the dominant frequencies.

Using a wider window length of m=600 and k=3 iterations, adaptively smoothed KZP algorithm was used to determine the spectrum for the simulated longitudinal dataset.

KZFT reconstruction of original signal embedded in the high noise of longitudinal observations ( missing rate 60%.)

Combinations of a few fixed frequency waves can complicate the recognition of the mixture of signals, but still remain predictable over time.

The reconstruction of these periodic signals of atmospheric tidal waves allows for an explanation and prediction of many anomalies present in extreme weather.

The estimation of the spectra of sunspot data using the DZ algorithm[3][6] provides two sharp frequency lines with periodicities close to 9.9 and 11.7 years.

An examination of the joint effect of multiple planets may reveal some long periods in sun activity and help explain climate fluctuations on earth.

It then examines these time intervals more carefully by reducing the window size so that the resolution of the smoothed outcome increases.

Figure 2 is a plot of a seasonal sine wave with amplitude of 1 unit, normally distributed noise (σ = 1), and a base signal with a break.

The application of an adaptive version of the KZ filter (KZA) finds the break as shown in Figure 5b.

The KZA algorithm has all of the typical advantages of a nonparametric approach; it does not require any specific model of the time series under investigation.

We may demonstrate application of 3D spatial KZ filter applied to the world records of temperature T(t, x, y) as a function of time t, longitude x and latitude y.

To select global climate fluctuations component parameters 25 month for time t, 3° for longitude and latitude were chosen for KZ filtration.

Standard average cosine square temperature distribution low[4] along latitudes were subtracted to identify fluctuations of climate in time and space.

Absolute humidity variable is keeping responsibility for major regional climate changings as it was displayed recently by Zurbenko Igor and Smith Devin in Kolmogorov–Zurbenko filters in spatiotemporal analysis.

KZ filter resolution performs exceptionally well compare to conventional methods and in fact is computationally optimal.

Andrey Kolmogorov and Igor Zurbenko on a research ship in the Pacific.
Figure 1.Transfer function of filter for k = 1.
Figure 2: Logarithm of the transfer function for the KZFT m , k filter with ν 0 = .04, m = 100, and k = 1 (black) or k = 5 (red).
Figure 3: The spectrum of the signal that is the summation of two sine waves about frequencies 0.08 and 0.10 cycles per unit time plus noise N(0,16) with 70% missing values. Adaptively smoothed KZP algorithm was used to determine the spectrum of the simulated data set.
Figure 4: The reconstructed signal that is the sum of two sine waves about frequencies 0.08 and 0.10 cycles per unit time from an original signal with added noise ~N(0, 16) and where 60% of the values were not available.
Figure 5a: Plot of signal + seasonality + noise. Figure 5b: KZA reconstruction of the signal with the break from the data in Figure 5a. Blue line is reconstruction of original signal as a black line.
Figure 6: Application of KZFT m , k to the data in Figure 5a. A regular low pass filter can not reproduce the break in the long term component.
Figure 7: Reconstruction of square image of the 2-dimensional signal of level 1 buried in the normal noise with σ=2. Left is noisy image, right is application to it of 2-dimensional KZA. Total field of display is 100x100 points, original image is 30x30 in the center.
Figure 8: Global Long-term Component in December 2007 KZ filter m = (3°, 3°, 25 months), k = 5, adjusted for latitude and altitude effects.