However, the above does not apply to §neural_networks since they are often so highly parameterized anyway, that they are effectively just as flexible as nonparametric models…
Classic Examples of non-parametric analogues to parametric models 🔗
|polynomial regression||Gaussian processes||function approx.|
|logistic regression||Gaussian process classifiers||classification|
|mixture models, k-means||Dirichlet process||mixtures clustering|
|hidden Markov models||infinite HMMs||time series|
|factor analysis / pPCA / PMF||infinite latent factor models||feature discovery|
Examples of parametric and nonparametric GANs 🔗
|Parametric (e.g. GMM)||Nonparametric (e.g. kNN, GP)|
|Strong assumption data distribution||Weaker assumption on data distribution.|
Parametric models assume the samples are from a specific distribution e.g. from a mixture of Gaussians where the number of Gaussian components is known a priori. This is restrictive since for most real-word problems we cannot know beforehand how complex the data is. For example, a nonparametric method should find the number of Gaussian components itself.
Theoretical equivalence 🔗
Neural networks can also be made nonparametric by taking the number of hidden units to infinity, in which case it turns out that such “infinite hidden layer” neural networks are equivalent to §Gaussian_processes with a certain covariance kernel. See Radford Neal’s 1994 PhD thesis, Bayesian Learning for Neural Networks.