Talk:Knowledge distillation: Difference between revisions
Content deleted Content added
Tag: |
Olle Gällmo (talk | contribs) →Knowledge capacity: new section |
||
| Line 3: | Line 3: | ||
{{WikiProject Statistics |importance=Low}} |
{{WikiProject Statistics |importance=Low}} |
||
}} |
}} |
||
== Knowledge capacity == |
|||
This article starts with the claim: |
|||
"While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized" |
|||
This is misleading. It suggests that deep networks have greater <u>storage </u>capacity. They don't. Hornik et al [1] showed already in 1989 that only one hidden layer is sufficient to approximate any function to any degree of accuracy. Deep learning may make it easier to <u>find</u> good representations, but that's not the same thing. |
|||
[1] Hornik, Stinchcombe & White, Multilayer feedforward networks are universal approximators, Neural Networks, vol.2, issue 5, pp. 359-366, 1989. https://www.sciencedirect.com/science/article/pii/0893608089900208 [[User:Olle Gällmo|Olle Gällmo]] ([[User talk:Olle Gällmo|talk]]) 11:59, 16 May 2024 (UTC) |
|||
Latest revision as of 11:59, 16 May 2024
 | This article is rated C-class on Wikipedia's content assessment scale. It is of interest to the following WikiProjects: | |||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||
Knowledge capacity[edit]
This article starts with the claim:
"While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized"
This is misleading. It suggests that deep networks have greater storage capacity. They don't. Hornik et al [1] showed already in 1989 that only one hidden layer is sufficient to approximate any function to any degree of accuracy. Deep learning may make it easier to find good representations, but that's not the same thing.
[1] Hornik, Stinchcombe & White, Multilayer feedforward networks are universal approximators, Neural Networks, vol.2, issue 5, pp. 359-366, 1989. https://www.sciencedirect.com/science/article/pii/0893608089900208 Olle Gällmo (talk) 11:59, 16 May 2024 (UTC)