AVT Statistical filtering algorithm

AVT Statistical filtering algorithm is an approach to improving quality of raw data collected from various sources. It is most effective in cases when there is inband noise present. In those cases AVT is better at filtering data then, band-pass filter or any digital filtering based on variation of.

Conventional filtering is useful when signal/data has different frequency than noise and signal/data is separated/filtered by frequency discrimination of noise. Frequency discrimination filtering is done using Low Pass, High Pass and Band Pass filtering which refers to relative frequency filtering criteria target for such configuration. Those filters are created using passive and active components and sometimes are implemented using software algorithms based on Fast Fourier transform (FFT).

AVT filtering is implemented in software and its inner working is based on statistical analysis of raw data.

When signal frequency/(useful data distribution frequency) coincides with noise frequency/(noisy data distribution frequency) we have inband noise. In this situations frequency discrimination filtering does not work since the noise and useful signal are indistinguishable and where AVT excels. To achieve filtering in such conditions there are several methods/algorithms available which are briefly described below.

Averaging algorithm

1. Collect n samples of data
2. Calculate average value of collected data
3. Present/record result as actual data

Median algorithm

1. Collect n samples of data
2. Sort the data in ascending or descending order. Note that order does not matter
3. Select the data that happen to be in n/2 position and present/record it as final result representing data sample

AVT algorithm

AVT inner working

AVT algorithm stands for Antonyan Vardan Transform and its implementation explained below.

1. Collect n samples of data
2. Calculate the standard deviation and average value
3. Drop any data that is greater or less than average ± one standard deviation
4. Calculate average value of remaining data
5. Present/record result as actual value representing data sample

This algorithm is based on amplitude discrimination and can easily reject any noise that is not like actual signal, otherwise statistically different then 1 standard deviation of the signal. Note that this type of filtering can be used in situations where the actual environmental noise is not known in advance. Notice that it is preferable to use the median in above steps than average. Originally the AVT algorithm used average value to compare it with results of median on the data window.

Filtering algorithms comparison

Using a system that has signal value of 1 and has noise added at 0.1% and 1% levels will simplify quantification of algorithm performance. The R script is used to create pseudo random noise added to signal and analyze the results of filtering using several algorithms. Please refer to "Reduce Inband Noise with the AVT Algorithm" article for details. This graphs show that AVT algorithm provides best results compared with Median and Averaging algorithms while using data sample size of 32, 64 and 128 values. Note that this graph was created by analyzing random data array of 10000 values. Sample of this data is graphically represented below.
From this graph it is apparent that AVT outperforms other filtering algorithms by providing 5% to 10% more accurate data when analyzing same datasets. Considering random nature of noise used in this numerical experiment that borderlines worst case situation where actual signal level is below ambient noise the precision improvements of processing data with AVT algorithm are significant.

AVT algorithm variations

Cascaded AVT

In some situations better results can be obtained by cascading several stages of AVT filtering. This will produce singular constant value which can be used for equipment that has known stable characteristics like thermometers, thermistors and other slow acting sensors.

Reverse AVT

1. Collect n samples of data
2. Calculate the standard deviation and average value
3. Drop any data that is within one standard deviation ± average band
4. Calculate average value of remaining data
5. Present/record result as actual data

This is useful for detecting minute signals that are close to background noise level.

Detecting Altered/modified Jupiter Picture with AVT image filter

Possible applications and uses

• Use to filter data that is near or below noise level
• Used in planet detection to filter out raw data from the Kepler space telescope
• Filter out noise from sound sources where all other filtering methods (Low-pass filter, High-pass filter, Band-pass filter, Digital filter) fail.
• Pre-process scientific data for data analysis (Smoothness) before plotting see (Plot (graphics))
• Used in SETI (Search for extraterrestrial intelligence) for detecting/distinguishing extraterrestrial signals from cosmic background
• Use AVT as image filtering algorithm to detect altered images. This image of Jupiter generated from this program, detecting alterations in original picture that was modified to be visually appealing by applying filters.

  

Another version of this comparison is the Reverse AVT filter applied to the same original Jupiter Image, where we only see that altered portion as Noise that was eliminated by AVT algorithm.

• Use AVT as image filtering algorithm to estimate data density from images. Picture of Pillars of Creation Nebula shows data density in filtered images from Hubble and Webb. Note that image on the left has big patches of missing data marked with simpler color patterns.

Pillars of creation picture data density difference detected with AVT comparing Webb to Hubble

References

1. Joseph, Favis; Balinadoa, C.; Paolo Dar Santos, Gerald; Escanilla, Rio; Darell C. Aguda, John; Ramona A. Alcantara, Ma.; Belen M. Roble, Mariela; F. Bueser, Jomalyn (May 5, 2020). "Design and implementation of water velocity monitoring system based on hydropower generation and antonyan vardan transform (AVT) statistics". 13Th International Engineering Research Conference (13Th Eureca 2019). Vol. 2233. p. 050003. doi:10.1063/5.0002323.
2. Vinicius, Cene; Mauricio, Tosin; J., Machado; A., Balbinot (April 2019). "Open Database for Accurate Upper-Limb Intent Detection Using Electromyography and Reliable Extreme Learning Machines". Sensors. 19 (8): 1864. Bibcode:2019Senso..19.1864C. doi:10.3390/s19081864. PMC 6515272. PMID 31003524.
3. HornCene, Vinicius; Balbinot, Alexandr (August 10, 2018), "Using the sEMG signal representativity improvement towards upper-limb movement classification reliability", Biomedical Signal Processing and Control, 46: 182–191, doi:10.1016/j.bspc.2018.07.014, ISSN 1746-8094, S2CID 52071917
4. Horn Cene, Vinicius; Ruschel dos Santos, Raphael; Balbinot, Alexandre (July 18, 2018). 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Honolulu, HI, USA: IEEE. pp. 5224–5227. doi:10.1109/EMBC.2018.8513468. ISBN 978-1-5386-3646-6.
5. AVT image Filtering algorithm in python