Noisy-Channel Model for Feature Extraction

Please enable javascript in your browser.

Hafez Kolahi, Hassan | 2022

90 Viewed

Type of Document: Ph.D. Dissertation
Language: Farsi
Document No: 55327 (19)
University: Sharif University of Technology
Department: Computer Engineering
Advisor(s): Kasaei, Shohreh; Soleymani Baghshah, Mahdieh
Abstract:
One of the approaches used in learning theory is using information theoretic tools. The general idea of this approach is that if we show the algorithm did not memorize the dataset, we could guarantee generalization. Noisy channel model is one of the important concepts in this approach. A noisy channel is a lossy process which maps the data to a compressed format.There are two ways to use noisy channel model in literature: input compression and model compression. One of the main results of this thesis is to show that the input compression methods can not explain the generalization of algorithms (despite previous belief). On the contrast by fixing some of the problems faced in the model compression literature, new results are proved in this part. In particular two approaches are introduced: conditioning and processing.Afterward, it was shown that not only the generalization gap can be explained by the proposed approach, but also the excess risk. For this a new view based on rate-distortion theory is introduced. Using this approach it was possible to find new bounds on minimum excess risk on both frequentist and Bayesian frameworks.
Keywords:
Information Theory ; Learning Theory ; Generalization ; Minimum Excess Risk ; Rate-Distortion Bound ; Feature Extraction

No TOC