Saliency map: Difference between revisions
Geysirhead (talk | contribs) mNo edit summary |
Geysirhead (talk | contribs) |
||
Line 14: | Line 14: | ||
*[[Seam carving|Image retargeting]]: It aims at resizing an image by expanding or shrinking the noninformative regions. Therefore, retargeting algorithms rely on the availability of saliency maps that accurately estimate all the salient image details.<ref>{{Cite journal|last1=Goferman|first1=Stas|last2=Zelnik-Manor|first2=Lihi|last3=Tal|first3=Ayellet|date=Oct 2012|title=Context-Aware Saliency Detection|url=https://ieeexplore.ieee.org/document/6112774|journal=IEEE Transactions on Pattern Analysis and Machine Intelligence|volume=34|issue=10|pages=1915–1926|doi=10.1109/TPAMI.2011.272|pmid=22201056 |issn=1939-3539}}</ref> |
*[[Seam carving|Image retargeting]]: It aims at resizing an image by expanding or shrinking the noninformative regions. Therefore, retargeting algorithms rely on the availability of saliency maps that accurately estimate all the salient image details.<ref>{{Cite journal|last1=Goferman|first1=Stas|last2=Zelnik-Manor|first2=Lihi|last3=Tal|first3=Ayellet|date=Oct 2012|title=Context-Aware Saliency Detection|url=https://ieeexplore.ieee.org/document/6112774|journal=IEEE Transactions on Pattern Analysis and Machine Intelligence|volume=34|issue=10|pages=1915–1926|doi=10.1109/TPAMI.2011.272|pmid=22201056 |issn=1939-3539}}</ref> |
||
*[[Object detection]] and [[Outline of object recognition|recognition]]: Instead of applying a computationally complex algorithm to the whole image, we can use it to the most salient regions of an image most likely to contain an object.<ref>{{Cite book|last1=Jiang|first1=Huaizu|last2=Wang|first2=Jingdong|last3=Yuan|first3=Zejian|last4=Wu|first4=Yang|last5=Zheng|first5=Nanning|last6=Li|first6=Shipeng|title=2013 IEEE Conference on Computer Vision and Pattern Recognition |chapter=Salient Object Detection: A Discriminative Regional Feature Integration Approach |date=June 2013|chapter-url=http://dx.doi.org/10.1109/cvpr.2013.271|pages=2083–2090 |publisher=IEEE|doi=10.1109/cvpr.2013.271|arxiv=1410.5926|isbn=978-0-7695-4989-7 }}</ref> |
*[[Object detection]] and [[Outline of object recognition|recognition]]: Instead of applying a computationally complex algorithm to the whole image, we can use it to the most salient regions of an image most likely to contain an object.<ref>{{Cite book|last1=Jiang|first1=Huaizu|last2=Wang|first2=Jingdong|last3=Yuan|first3=Zejian|last4=Wu|first4=Yang|last5=Zheng|first5=Nanning|last6=Li|first6=Shipeng|title=2013 IEEE Conference on Computer Vision and Pattern Recognition |chapter=Salient Object Detection: A Discriminative Regional Feature Integration Approach |date=June 2013|chapter-url=http://dx.doi.org/10.1109/cvpr.2013.271|pages=2083–2090 |publisher=IEEE|doi=10.1109/cvpr.2013.271|arxiv=1410.5926|isbn=978-0-7695-4989-7 }}</ref> |
||
*[[Explainable Artificial Intelligence]] in the context of vision models: Saliency maps are a prominent tool in XAI,<ref name=mueller24>{{cite journal |last1=Müller |first1=Romy |title=How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency maps |journal=Arxiv |date=2024 |doi=10.48550/arXiv.2404.16042}}</ref> providing visual explanations of the decision-making process of [[machine learning model]]s, particularly [[deep neural network]]s. These maps highlight the regions in input images, text, or other types of data that are most influential in the model's output, effectively indicating where the model is "looking" when making a prediction. By illustrating which parts of the input are deemed important, saliency maps help in understanding the internal workings of otherwise black box models, thereby fostering trust and transparency. In image classification tasks, for example, saliency maps can identify pixels or regions that contribute most to a specific class decision. One of the most prominent techniques to generate saliency maps is [[Gradient-weighted Class Activation Mapping]] (Grad-CAM), which is cited more than 20,000 times.<ref name=mueller24/> |
|||
===Saliency as a segmentation problem=== |
===Saliency as a segmentation problem=== |
Revision as of 10:49, 26 May 2024
![](http://upload.wikimedia.org/wikipedia/commons/thumb/1/1a/Saliencymap_example.jpg/476px-Saliencymap_example.jpg)
In computer vision, a saliency map is an image that highlights either the region on which people's eyes focus first or the most relevant regions for vision models.[1] The goal of a saliency map is to reflect the degree of importance of a pixel to the human visual system or a black box computer vision model.
For example, in this image, a person first looks at the fort and light clouds, so they should be highlighted on the saliency map. Saliency maps engineered in artificial or computer vision are typically not the same as the actual saliency map constructed by biological or natural vision.
Application
Overview
Saliency maps have applications in a variety of different problems. Some general applications:
- Image and video compression: The human eye focuses only on a small region of interest in the frame. Therefore, it is not necessary to compress the entire frame with uniform quality. According to the authors, using a salience map reduces the final size of the video with the same visual perception.[2]
- Image and video quality assessment: The main task for an image or video quality metric is a high correlation with user opinions. Differences in salient regions are given more importance and thus contribute more to the quality score.[3]
- Image retargeting: It aims at resizing an image by expanding or shrinking the noninformative regions. Therefore, retargeting algorithms rely on the availability of saliency maps that accurately estimate all the salient image details.[4]
- Object detection and recognition: Instead of applying a computationally complex algorithm to the whole image, we can use it to the most salient regions of an image most likely to contain an object.[5]
- Explainable Artificial Intelligence in the context of vision models: Saliency maps are a prominent tool in XAI,[6] providing visual explanations of the decision-making process of machine learning models, particularly deep neural networks. These maps highlight the regions in input images, text, or other types of data that are most influential in the model's output, effectively indicating where the model is "looking" when making a prediction. By illustrating which parts of the input are deemed important, saliency maps help in understanding the internal workings of otherwise black box models, thereby fostering trust and transparency. In image classification tasks, for example, saliency maps can identify pixels or regions that contribute most to a specific class decision. One of the most prominent techniques to generate saliency maps is Gradient-weighted Class Activation Mapping (Grad-CAM), which is cited more than 20,000 times.[6]
Saliency as a segmentation problem
Saliency estimation may be viewed as an instance of image segmentation. In computer vision, image segmentation is the process of partitioning a digital image into multiple segments (sets of pixels, also known as superpixels). The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze. Image segmentation is typically used to locate objects and boundaries (lines, curves, etc.) in images. More precisely, image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain characteristics.[7]
Algorithms
Overview
There are three forms of classic saliency estimation algorithms implemented in OpenCV:
- Static saliency: Relies on image features and statistics to localize the regions of interest of an image.
- Motion saliency: Relies on motion in a video, detected by optical flow. Objects that move are considered salient.
- Objectness: Objectness reflects how likely an image window covers an object. These algorithms generate a set of bounding boxes of where an object may lie in an image.
In addition to classic approaches, neural-network-based are also popular. There are examples of neural networks for motion saliency estimation:
- TASED-Net: It consists of two building blocks. First, the encoder network extracts low-resolution spatiotemporal features, and then the following prediction network decodes the spatially encoded features while aggregating all the temporal information.
- STRA-Net: It emphasizes two essential issues. First, spatiotemporal features integrated via appearance and optical flow coupling, and then multi-scale saliency learned via attention mechanism.
- STAViS: It combines spatiotemporal visual and auditory information. This approach employs a single network that learns to localize sound sources and to fuse the two saliencies to obtain a final saliency map.
Example implementation
First, we should calculate the distance of each pixel to the rest of pixels in the same frame:
is the value of pixel , in the range of [0,255]. The following equation is the expanded form of this equation.
- SALS(Ik) = |Ik - I1| + |Ik - I2| + ... + |Ik - IN|
Where N is the total number of pixels in the current frame. Then we can further restructure our formula. We put the value that has same I together.
- SALS(Ik) = Σ Fn × |Ik - In|
Where Fn is the frequency of In. And the value of n belongs to [0,255]. The frequencies is expressed in the form of histogram, and the computational time of histogram is time complexity.
Time complexity
This saliency map algorithm has time complexity. Since the computational time of histogram is time complexity which N is the number of pixel's number of a frame. Besides, the minus part and multiply part of this equation need 256 times operation. Consequently, the time complexity of this algorithm is which equals to .
Pseudocode
All of the following code is pseudo MATLAB code. First, read data from video sequences.
for k = 2 : 1 : 13 % which means from frame 2 to 13, and in every loop K's value increase one.
I = imread(currentfilename); % read current frame
I1 = im2single(I); % convert double image into single(requirement of command vlslic)
l = imread(previousfilename); % read previous frame
I2 = im2single(l);
regionSize = 10; % set the parameter of SLIC this parameter setting are the experimental result. RegionSize means the superpixel size.
regularizer = 1; % set the parameter of SLIC
segments1 = vl_slic(I1, regionSize, regularizer); % get the superpixel of current frame
segments2 = vl_slic(I2, regionSize, regularizer); % get superpixel of the previous frame
numsuppix = max(segments1(:)); % get the number of superpixel all information about superpixel is in this link [http://www.vlfeat.org/overview/slic.html]
regstats1 = regionprops(segments1, ’all’);
regstats2 = regionprops(segments2, ’all’); % get the region characteristic based on segments1
After we read data, we do superpixel process to each frame. Spnum1 and Spnum2 represent the pixel number of current frame and previous pixel.
% First, we calculate the value distance of each pixel.
% This is our core code
for i = 1:1:spnum1 % From the first pixel to the last one. And in every loop i++
for j = 1:1:spnum2 % From the first pixel to the last one. j++. previous frame
centredist(i:j) = sum((center(i) - center(j))); % calculate the center distance
end
end
Then we calculate the color distance of each pixel, this process we call it contract function.
for i = 1:1:spnum1 % From first pixel of current frame to the last one pixel. I ++
for j = 1:1:spnum2 % From first pixel of previous frame to the last one pixel. J++
posdiff(i, j) = sum((regstats1(j).Centroid’ - mupwtd(:, i))); % Calculate the color distance.
end
end
After this two process, we will get a saliency map, and then store all of these maps into a new FileFolder.
Difference in algorithms
The major difference between function one and two is the difference of contract function. If spnum1 and spnum2 both represent the current frame's pixel number, then this contract function is for the first saliency function. If spnum1 is the current frame's pixel number and spnum2 represent the previous frame's pixel number, then this contract function is for second saliency function. If we use the second contract function which using the pixel of the same frame to get center distance to get a saliency map, then we apply this saliency function to each frame and use current frame's saliency map minus previous frame's saliency map to get a new image which is the new saliency result of the third saliency function.
![](http://upload.wikimedia.org/wikipedia/commons/thumb/7/7d/Wiki102.png/220px-Wiki102.png)
Datasets
The saliency dataset usually contains human eye movements on some image sequences. It is valuable for new saliency algorithm creation or benchmarking the existing one. The most valuable dataset parameters are spatial resolution, size, and eye-tracking equipment. Here is part of the large datasets table from MIT/Tübingen Saliency Benchmark datasets, for example.
Dataset | Resolution | Size | Observers | Durations | Eyetracker |
---|---|---|---|---|---|
CAT2000 | 1920×1080px | 4000 images | 24 | 5 sec | EyeLink 1000 (1000Hz) |
EyeTrackUAV2 | 1280×720px | 43 videos | 30 | 33 sec | EyeLink 1000 Plus (1000 Hz, binocular) |
CrowdFix | 1280×720px | 434 videos | 26 | 1-3 sec | The Eyetribe Eyetracker (60 Hz) |
SAVAM | 1920×1080px | 43 videos | 50 | 20 sec | SMI iViewXTM Hi-Speed 1250 (500Hz) |
To collect a saliency dataset, image or video sequences and eye-tracking equipment must be prepared, and observers must be invited. Observers must have normal or corrected to normal vision and must be at the same distance from the screen. At the beginning of each recording session, the eye-tracker recalibrates. To do this, the observer fixates his gaze on the screen center. Then the session started, and saliency data are collected by showing sequences and recording eye gazes.
The eye-tracking device is a high-speed camera, capable of recording eye movements at least 250 frames per second. Images from the camera are processed by the software, running on a dedicated computer returning gaze data.
References
- ^ Subhash, Bijil (6 March 2022). "Explainable AI: Saliency Maps". Medium. Retrieved 26 May 2024.
- ^ Guo, Chenlei; Zhang, Liming (Jan 2010). "A Novel Multiresolution Spatiotemporal Saliency Detection Model and Its Applications in Image and Video Compression". IEEE Transactions on Image Processing. 19 (1): 185–198. Bibcode:2010ITIP...19..185G. doi:10.1109/TIP.2009.2030969. ISSN 1057-7149. PMID 19709976. S2CID 1154218.
- ^ Tong, Yubing; Konik, Hubert; Cheikh, Faouzi; Tremeau, Alain (2010-05-01). "Full Reference Image Quality Assessment Based on Saliency Map Analysis". Journal of Imaging Science and Technology. 54 (3): 30503–1–30503-14. doi:10.2352/J.ImagingSci.Technol.2010.54.3.030503. hdl:11250/142490.
- ^ Goferman, Stas; Zelnik-Manor, Lihi; Tal, Ayellet (Oct 2012). "Context-Aware Saliency Detection". IEEE Transactions on Pattern Analysis and Machine Intelligence. 34 (10): 1915–1926. doi:10.1109/TPAMI.2011.272. ISSN 1939-3539. PMID 22201056.
- ^ Jiang, Huaizu; Wang, Jingdong; Yuan, Zejian; Wu, Yang; Zheng, Nanning; Li, Shipeng (June 2013). "Salient Object Detection: A Discriminative Regional Feature Integration Approach". 2013 IEEE Conference on Computer Vision and Pattern Recognition. IEEE. pp. 2083–2090. arXiv:1410.5926. doi:10.1109/cvpr.2013.271. ISBN 978-0-7695-4989-7.
- ^ a b Müller, Romy (2024). "How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency maps". Arxiv. doi:10.48550/arXiv.2404.16042.
- ^ A. Maity (2015). "Improvised Salient Object Detection and Manipulation". arXiv:1511.02999 [cs.CV].
External links
- Zhai, Yun; Shah, Mubarak (2006-10-23). "Visual attention detection in video sequences using spatiotemporal cues". Proceedings of the 14th ACM international conference on Multimedia. MM '06. New York, NY, USA: ACM. pp. 815–824. CiteSeerX 10.1.1.80.4848. doi:10.1145/1180639.1180824. ISBN 978-1595934475. S2CID 5219826.
- VLfeat: http://www.vlfeat.org/index.html
- Saliency map at Scholarpedia