Mean shift is a non-parametric feature-space analysis technique for locating the maxima of a density function, a so-called mode-seeking algorithm. Application domains include cluster analysis in computer vision and image processing.
The mean shift procedure was originally presented in 1975 by Fukunaga and Hostetler.
Mean shift is a procedure for locating the maxima of a density function given discrete data sampled from that function. It is useful for detecting the modes of this density. This is an iterative method, and we start with an initial estimate . Let a kernel function be given. This function determines the weight of nearby points for re-estimation of the mean. Typically a Gaussian kernel on the distance to the current estimate is used, . The weighted mean of the density in the window determined by
where is the neighborhood of , a set of points for which .
The mean-shift algorithm now sets , and repeats the estimation until converges.
Let data be a finite set S embedded in the n-dimensional Euclidean space, X. Let K be a flat kernel that is the characteristic function of the -ball in X,
The difference is called mean shift in Fukunaga and Hostetler. The repeated movement of data points to the sample means is called the mean shift algorithm. In each iteration of the algorithm, is performed for all simultaneously. The first question, then, is how to estimate the density function given a sparse set of samples. One of the simplest approaches is to just smooth the data, e.g., by convolving it with a fixed kernel of width ,
where are the input samples and is the kernel function (or Parzen window). h is the only paramter in the algorithm and is called the bandwidth. This approach is known as kernel density estimation or the Parzen window technique. Once we have computed from equation above, we can find its local maxima using gradient ascent or some other optimization technique. The problem with this "brute force" approach is that, for higher dimensions, it becomes computationally prohibitive to evaluate over the complete search space. Instead, mean shift uses a variant of what is known in the optimization literature as multiple restart gradient descent. Starting at some guess for a local maximum, , which can be a random input data point , mean shift computes the gradient of the density estimate at and takes an uphill step in that direction.
Types of kernels
Kernel definition: Let X be the n-dimensional Euclidean space, . Denote the ith component of x by . The norm of x is a non-negative number. A function K: is said to be a kernel if there exists a profile, , such that
- k is non-negative.
- k is nonincreasing: if .
- k is piecewise continuous and
The two frequently used kernels for mean shift are:
- Flat kernel
- Gaussian kernel
where , the normalization constant, makes G(x) integrate to one and is called the profile of the kernel. It simplifies calculation in the case of multivariate data. The profile of the Gaussian kernel is: and therefore, the multivariate Gaussian kernel with the standard deviation , will be: where d is the number of dimensions. It's also worth mentioning that the standard deviation for the Gaussian kernel works as the bandwidth parameter,
Consider a set of points in two-dimensional space. Assume a circular window centered at C and having radius r as the kernel. Mean shift is a hill climbing algorithm which involves shifting this kernel iteratively to a higher density region until convergence. Every shift is defined by a mean shift vector. The mean shift vector always points toward the direction of the maximum increase in the density. At every iteration the kernel is shifted to the centroid or the mean of the points within it. The method of calculating this mean depends on the choice of the kernel. In this case if a Gaussian kernel is chosen instead of a flat kernel, then every point will first be assigned a weight which will decay exponentially as the distance from the kernel's center increases. At convergence, there will be no direction at which a shift can accommodate more points inside the kernel.
The mean shift algorithm can be used for visual tracking. The simplest such algorithm would create a confidence map in the new image based on the color histogram of the object in the previous image, and use mean shift to find the peak of a confidence map near the object's old position. The confidence map is a probability density function on the new image, assigning each pixel of the new image a probability, which is the probability of the pixel color occurring in the object in the previous image. A few algorithms, such as ensemble tracking, CAMshift, expand on this idea.
Let and be the d-dimensional input and filtered image pixels in the joint spatial-range domain. For each pixel,
- Initialize and
- Compute according to until convergence, .
- Assign . The superscripts s and r denote the spatial and range components of a vector, respectively. The assignment specifies that the filtered data at the spatial location axis will have the range component of the point of convergence .
- Mean shift is an application-independent tool suitable for real data analysis.
- Does not assume any predefined shape on data clusters.
- It is capable of handling arbitrary feature spaces.
- The procedure relies on choice of a single parameter: bandwidth.
- The bandwidth/window size 'h' has a physical meaning, unlike k-means.
- The selection of a window size is not a trivial trick.
- Inappropriate window size can cause modes to be merged, or generate additional “shallow” modes.
- Often requires using adaptive window size.
Mean shift and k-means clustering
The mean shift clustering algorithm has two main drawbacks. Firstly, the algorithm is pretty calculation intensive; it requires in general operations, where N is the number of data points and k is the number of average iteration steps for each data point. Secondly, the mean shift algorithm relies on sufficient high data density with clear gradient to locate the cluster centers. In particular, the main shift algorithm often fails to find appropriate clusters for so called data outliers, or those data points located between natural clusters.
The k-means algorithm does not have the above two problems. The k-means algorithm normally requires only operations, so that the k-means algorithm can be applied to relatively large dataset. However, k-means has two significant limitations. Firstly, the k-means algorithm requires that the number of clusters to be pre-determined. In practise, it is often difficult to find the right cluster number, so that we often just pick a sufficiently large cluster number. This will result in situations that some natural clusters will be represented by multiple clusters found by the k-means algorithm. Secondly, the k-means algorithm is in general incapable to find non-convex clusters. The second limitation makes the k-means algorithm inadequate for complex non-linear data. These problems can be overcome, by simply combining the two algorithms mean shift and k-means together.
- Cheng, Yizong (August 1995). "Mean Shift, Mode Seeking, and Clustering". IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE) 17 (8): 790–799. doi:10.1109/34.400568.
- Comaniciu, Dorin; Peter Meer (May 2002). "Mean Shift: A Robust Approach Toward Feature Space Analysis". IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE) 24 (5): 603–619. doi:10.1109/34.1000236.
- Fukunaga, Keinosuke; Larry D. Hostetler (January 1975). "The Estimation of the Gradient of a Density Function, with Applications in Pattern Recognition". IEEE Transactions on Information Theory (IEEE) 21 (1): 32–40. doi:10.1109/TIT.1975.1055330. Retrieved 2008-02-29.
- Avidan, Shai (2005). "Ensemble Tracking". 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) (San Diego, California: IEEE) 2. ISBN 0-7695-2372-2.
- Emami, Ebrahim (2013). "Online failure detection and correction for CAMShift tracking algorithm". 2013 Iranian Conference on Machine Vision and Image Processing (MVIP) (IEEE) 2: 180–183.
- Li, James (2012). "Visualizing High Dimensional Data".
- Scikit-learn library Numpy/Python implementation uses ball tree for efficient neighboring points lookup
- EDISON library. C++ implementation of mean-shift-based image segmentation. There is also a Matlab interface for EDISON.
- OpenCV contains mean-shift implementation via cvMeanShift Method
- Aiphial. Java-based mean-shift implementation for numeric data clustering and image segmentation
- Apache Mahout. An map-reduce based implementation of MeanShift clustering written on Apache Hadoop.
- CAMSHIFT project. A MATLAB implementation of CAMSHIFT algorithm.
- OTB MeanShift. A C++ implementation using the Orfeo Toolbox.
- ImageJ Plug-in. Image filtering using the mean shift filter.
- Mean-shift google code. An simple implementation of mean-shift as image filtering tool.
- A lesson from Prof. M. Shah on this topic