Error tolerance (PAC learning)
|Machine learning and|
Error tolerance (PAC learning)
In PAC learning, error tolerance refers to the ability of an algorithm to learn when the examples received have been corrupted in some way. In fact, this is a very common and important issue since in many applications it is not possible to access noise-free data. Noise can interfere with the learning process at different levels: the algorithm may receive data that have been occasionally mislabeled, or the inputs may have some false information, or the classification of the examples may have been maliciously adulterated.
Notation and the Valiant learning model
In the following, let be our -dimensional input space. Let be a class of functions that we wish to use in order to learn a -valued target function defined over . Let be the distribution of the inputs over . The goal of a learning algorithm is to choose the best function such that it minimizes . Let us suppose we have a function that can measure the complexity of . Let be an oracle that, whenever called, returns an example and its correct label .
Definition: We say that is efficiently learnable using in the Valiant setting if there exists a learning algorithm that has access to and a polynomial such that for any and it outputs, in a number of calls to the oracle bounded by , a function that satisfies with probability at least the condition .
In the classification noise model a noise rate is introduced. Then, instead of that returns always the correct label of example , algorithm can only call a faulty oracle that will flip the label of with probability . As in the Valiant case, the goal of a learning algorithm is to choose the best function such that it minimizes . In applications it is difficult to have access to the real value of , but we assume we have access to its upperbound . Note that if we allow the noise rate to be , then learning becomes impossible in any amount of computation time, because every label conveys no information about the target function.
Definition: We say that is efficiently learnable using in the classification noise model if there exists a learning algorithm that has access to and a polynomial such that for any , and it outputs, in a number of calls to the oracle bounded by , a function that satisfies with probability at least the condition .
Statistical query learning
Statistical Query Learning is a kind of active learning problem in which the learning algorithm can decide if to request information about the likelihood that a function correctly labels example , and receives an answer accurate within a tolerance . Formally, whenever the learning algorithm calls the oracle , it receives as feedback probability , such that .
Definition: We say that is efficiently learnable using in the statistical query learning model if there exists a learning algorithm that has access to and polynomials , , and such that for any the following hold:
- can evaluate in time ;
- is bounded by
- outputs a model such that , in a number of calls to the oracle bounded by .
Note that the confidence parameter does not appear in the definition of learning. This is because the main purpose of is to allow the learning algorithm a small probability of failure due to an unrepresentative sample. Since now always guarantees to meet the approximation criterion , the failure probability is no longer needed.
The statistical query model is strictly weaker than the PAC model: any efficiently SQ-learnable class is efficiently PAC learnable in the presence of classification noise, but there exist efficient PAC-learnable problems such as parity that are not efficiently SQ-learnable.
In the malicious classification model an adversary generates errors to foil the learning algorithm. This setting describes situations of error burst, which may occur when for a limited time transmission equipment malfunctions repeatedly. Formally, algorithm calls an oracle that returns a correctly labeled example drawn, as usual, from distribution over the input space with probability , but it returns with probability an example drawn from a distribution that is not related to . Moreover, this maliciously chosen example may strategically selected by an adversary who has knowledge of , , , or the current progress of the learning algorithm.
Definition: Given a bound for , we say that is efficiently learnable using in the malicious classification model, if there exist a learning algorithm that has access to and a polynomial such that for any , it outputs, in a number of calls to the oracle bounded by , a function that satisfies with probability at least the condition .
Errors in the inputs: nonuniform random attribute noise
This type of error can irreparably foil the algorithm, in fact the following theorem holds:
In the nonuniform random attribute noise setting, an algorithm can output a function such that only if .
- Valiant, L. G. (August 1985). Learning Disjunction of Conjunctions. In IJCAI (pp. 560–566).
- Valiant, Leslie G. "A theory of the learnable." Communications of the ACM 27.11 (1984): 1134–1142.
- Laird, P. D. (1988). Learning from good and bad data. Kluwer Academic Publishers.
- Kearns, Michael. "Efficient noise-tolerant learning from statistical queries." Journal of the ACM 45.6 (1998): 983–1006.
- Brunk, Clifford A., and Michael J. Pazzani. "An investigation of noise-tolerant relational concept learning algorithms." Proceedings of the 8th International Workshop on Machine Learning. 1991.
- Kearns, M. J., & Vazirani, U. V. (1994). An introduction to computational learning theory, chapter 5. MIT press.
- Angluin, D., & Laird, P. (1988). Learning from noisy examples. Machine Learning, 2(4), 343–370.
- Kearns, M. (1998). [www.cis.upenn.edu/~mkearns/papers/sq-journal.pdf Efficient noise-tolerant learning from statistical queries]. Journal of the ACM, 45(6), 983–1006.
- Kearns, M., & Li, M. (1993). [www.cis.upenn.edu/~mkearns/papers/malicious.pdf Learning in the presence of malicious errors]. SIAM Journal on Computing, 22(4), 807–837.
- Goldman, S. A., & Robert, H. (1991). Sloan. The difficulty of random attribute noise. Technical Report WUCS 91 29, Washington University, Department of Computer Science.
- Sloan, R. H. (1989). Computational learning theory: New models and algorithms (Doctoral dissertation, Massachusetts Institute of Technology).