Weak supervision

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search

Weak supervision is a branch of machine learning where noisy, limited, or imprecise sources are used to provide supervision signal for labeling large amounts of training data in a supervised learning setting.[1] This approach alleviates the burden of obtaining hand-labeled data sets, which can be costly or impractical. Instead, inexpensive weak labels are employed with the understanding that they are imperfect, but can nonetheless be used to create a strong predictive model.[2][3][4][5] Weak supervision is a generalization of semi-supervision, which is the historical approach to address the greediness of supervised learning in term of human annotations.

Problem of labeled training data[edit]

Machine learning models and techniques are increasingly accessible to researchers and developers; the real-world usefulness of these models, however, depends on access to high-quality labeled training data.[6] This need for labeled training data often proves to be a significant obstacle to the application of machine learning models within an organization or industry.[1] This bottleneck effect manifests itself in various ways, including the following examples:

Insufficient quantity of labeled data

When machine learning techniques are initially used in new applications or industries, there is often not enough training data available to apply traditional processes.[7] Some industries have the benefit of decades' worth of training data readily available; those that do not are at a significant disadvantage. In such cases, obtaining training data may be impractical, expensive, or impossible without waiting years for its accumulation.

Insufficient subject-matter expertise to label data

When labeling training data requires specific relevant expertise, creation of a usable training data set can quickly become prohibitively expensive.[7] This issue is likely to occur, for example, in biomedical or security-related applications of machine learning.

Insufficient time to label and prepare data

Most of the time required to implement machine learning is spent in preparing data sets.[7] When an industry or research field deals with problems that are, by nature, rapidly evolving, it can be impossible to collect and prepare data quickly enough for results to be useful in real-world applications. This issue could occur, for example, in fraud detection or cybersecurity applications.

Other areas of machine learning exist that are likewise motivated by the demand for increased quantity and quality of labeled training data but employ different high-level techniques to approach this demand. These other approaches include active learning, semi-supervised learning, and transfer learning.[1]

Types of weak labels[edit]

Weak labels are intended to decrease the cost and increase the efficiency of human efforts expended in hand-labeling data. They can take many forms, including the following:

  • Imprecise or inexact labels: developers may use higher-level, less precise input from subject-matter experts to create heuristic rules, define expected distributions, or impose other constraints on the training data.[8][3][9][10]
  • Inaccurate labels: developers may use inexpensive, lower-quality input through means such as crowdsourcing to obtain labels that are numerous, but not expected to be perfectly correct.[3][2]
  • Existing resources: developers may take advantage of existing resources (such as knowledge bases, alternative data sets, or pre-trained models[1]) to create labels that are helpful, though not perfectly suited for the given task.[3]

Applications of weak supervision[edit]

Applications of weak supervision are numerous and varied within the machine learning research community.

In 2014, researchers from UC Berkeley made use of the principles of weak supervision to propose an iterative learning algorithm that solely depends on labels generated by heuristics and alleviates the need of collecting any ground-truth labels.[11][12] The algorithm was applied to smart meter data to learn about the household's occupancy without ever asking for the occupancy data, which has raised issues of privacy and security as covered by an article in IEEE Spectrum.[13]

In 2018, researchers from UC Riverside proposed a method to localize actions/events in videos using only weak supervision, i.e., video-level labels, without any information about the start and end time of the events while training. Their work [14] introduced an attention-based similarity between two videos, which acts as a regularizer for learning with weak labels. Thereafter in 2019, they introduced a new problem [15] of event localization in videos using text queries from users, but with weak annotations while training. Later in a collaboration with NEC Laboratories America a similar attention-based alignment mechanism with weak labels was introduced for adapting a source semantic segmentation model to a target domain.[16] When the weak labels of the target images are estimated using the source model, it is unsupervised domain adaptation, requiring no target annotation cost, and when the weak labels are acquired from an annotator, it incurs a very small amount of annotation cost and falls under the category of weakly-supervised domain adaptation, which is first introduced in this work for semantic segmentation.

Stanford University researchers created Snorkel, an open-source system for quickly assembling training data through weak supervision.[17] Snorkel employs the central principles of the data programming paradigm,[8] in which developers create labeling functions, which are then used to programmatically label data, and employs supervised learning techniques to assess the accuracy of those labeling functions.[18] In this way, potentially low-quality inputs can be used to create high-quality models.

In a joint work with Google, Stanford researchers showed that existing organizational knowledge resources could be converted into weak supervision sources and used to significantly decrease development costs and time.[19]

In 2019, Massachusetts Institute of Technology and Google researchers released cleanlab, the first standardized Python package for machine learning and deep learning with noisy labels.[20] Cleanlab implements confident learning,[21][22] a framework of theory and algorithms for dealing with uncertainty in dataset labels, to (1) find label errors in datasets, (2) characterize label noise, and (3) standardize and simplify research in weak supervision and learning with noisy labels.[23]

Researchers at University of Massachusetts Amherst propose augmenting traditional active learning approaches by soliciting labels on features rather than instances within a data set.[24]

Researchers at Johns Hopkins University propose reducing the cost of labeling data sets by having annotators provide rationales supporting each of their data annotations, then using those rationales to train both discriminative and generative models for labeling additional data.[25]

Researchers at University of Alberta propose a method that applies traditional active learning approaches to enhance the quality of the imperfect labels provided by weak supervision.[26]

  1. ^ a b c d Alex Ratner, Stephen Bach, Paroma Varma, Chris Ré And referencing work by many other members of Hazy Research. "Weak Supervision: The New Programming Paradigm for Machine Learning". hazyresearch.github.io. Retrieved 2019-06-05.CS1 maint: multiple names: authors list (link)
  2. ^ a b Campagner, Andrea; Ciucci, Davide; Svensson, Carl Magnus; Figge, Marc Thilo; Cabitza, Federico (2021). "Ground truthing from multi-rater labeling with three-way decision and possibility theory". Information Sciences. 545: 771–790. doi:10.1016/j.ins.2020.09.049.
  3. ^ a b c d Zhou, Zhi-Hua (2018). "A Brief Introduction to Weakly Supervised Learning" (PDF). National Science Review. 5: 44–53. doi:10.1093/NSR/NWX106. S2CID 44192968. Archived from the original (PDF) on 22 February 2019. Retrieved 4 June 2019.
  4. ^ Nodet, Pierre; Lemaire, Vincent; Bondu, Alexis; Cornuéjols, Antoine; Ouorou, Adam (2020-12-20). "From Weakly Supervised Learning to Biquality Learning, a brief introduction". arXiv:2012.09632 [cs.LG].
  5. ^ Cabannes, Vivien; Rudi, Alessandro; Bach, Francis (2021). "Disambiguation of weak supervision with exponential convergence rates". ICML. arXiv:2102.02789.
  6. ^ "Datasets Over Algorithms". Space Machine. Retrieved 2019-06-05.
  7. ^ a b c Roh, Yuji (8 Nov 2018). "A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective". arXiv:1811.03402 [cs.LG].
  8. ^ a b Ré, Christopher; Selsam, Daniel; Wu, Sen; De Sa, Christopher; Ratner, Alexander (2016-05-25). "Data Programming: Creating Large Training Sets, Quickly". arXiv:1605.07723v3 [stat.ML].
  9. ^ Cabannes, Vivien; Rudi, Alessandro; Bach, Francis (2020). "Structured Prediction with Partial Labelling through the Infimum Loss". ICML. 37. arXiv:2003.00920.
  10. ^ Hüllermeier, Eyke (2014). "Learning from imprecise and fuzzy observations: Data disambiguation through generalized loss minimization". International Journal of Approximate Reasoning. 55: 1519–1534. arXiv:1305.0698. doi:10.1016/j.ijar.2013.09.003.
  11. ^ Jin, Ming; Jia, Ruoxi; Kang, Zhaoyi; Konstantakopoulos, Ioannis; Spanos, Costas (2014). "PresenceSense: zero-training algorithm for individual presence detection based on power monitoring". Proceedings of the 1st ACM Conference on Embedded Systems for Energy-Efficient Buildings: 1–10. doi:10.1145/2674061.2674073.
  12. ^ Jin, Ming; Jia, Ruoxi; Spanos, Costas (2017). "Virtual occupancy sensing: using smart meters to indicate your presence". IEEE Transactions on Mobile Computing. 16 (11): 3264–3277. arXiv:1407.4395. doi:10.1109/TMC.2017.2684806.
  13. ^ "What does smart meter know about you?". IEEE Spectrum.
  14. ^ Paul, Sujoy; Roy, Sourya; Roy-Chowdhury, Amit K. (2018). "W-TALC: Weakly-supervised Temporal Activity Localization and Classification". European Conference on Computer Vision (ECCV). arXiv:1807.10418.
  15. ^ Mithun, Niluthpol Chowdhury; Paul, Sujoy; Roy-Chowdhury, Amit K. (2019). "Weakly Supervised Video Moment Retrieval From Text Queries". Computer Vision and Pattern Recognition (CVPR). arXiv:1904.03282.
  16. ^ Paul, Sujoy; Tsai, Yi-Hsuan; Schulter, Samuel; Roy-Chowdhury, Amit K.; Chandraker, Manmohan (2020). "Domain Adaptive Semantic Segmentation Using Weak Labels". European Conference on Computer Vision (ECCV). arXiv:2007.15176.
  17. ^ "Snorkel and The Dawn of Weakly Supervised Machine Learning · Stanford DAWN". dawn.cs.stanford.edu. Retrieved 2019-06-05.
  18. ^ "Snorkel by HazyResearch". hazyresearch.github.io. Retrieved 2019-06-05.
  19. ^ Malkin, Rob; Ré, Christopher; Kuchhal, Rahul; Alborzi, Houman; Hancock, Braden; Ratner, Alexander; Sen, Souvik; Xia, Cassandra; Shao, Haidong (2018-12-02). "Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale". Proceedings. Acm-Sigmod International Conference on Management of Data. 2019: 362–375. arXiv:1812.00417. Bibcode:2018arXiv181200417B. doi:10.1145/3299869.3314036. PMC 6879379. PMID 31777414.
  20. ^ "Announcing cleanlab: a Python Package for ML and Deep Learning on Datasets with Label Errors". l7.curtisnorthcutt.com. Retrieved 2020-02-04.
  21. ^ "An Introduction to Confident Learning: Finding and Learning with Label Errors in Datasets". l7.curtisnorthcutt.com. Retrieved 2020-02-04.
  22. ^ Northcutt, Curtis G.; Jiang, Lu; Chuang, Isaac L. (2019-10-31). "Confident Learning: Estimating Uncertainty in Dataset Labels". arXiv:1911.00068 [stat.ML].
  23. ^ Northcutt, Curtis. "CleanLab for Finding and Learning with Noisy Labels". Retrieved 9 October 2019.
  24. ^ Druck, Gregory. "Active Learning by Labeling Features" (PDF). Retrieved 4 June 2019.
  25. ^ Zaidan, Omar. "Machine Learning with Annotator Rationales to Reduce Annotation Cost" (PDF). Retrieved 4 June 2019.
  26. ^ Nashaat, Mona; Ghosh, Aindrila; Miller, James; Quader, Shaikh; Marston, Chad; Puget, Jean-Francois (December 2018). "Hybridization of Active Learning and Data Programming for Labeling Large Industrial Datasets". 2018 IEEE International Conference on Big Data (Big Data). Seattle, WA, USA: IEEE: 46–55. doi:10.1109/BigData.2018.8622459. ISBN 9781538650356. S2CID 59233854.