Figure 1: From Ghahramani’s slides on GPs and NNs (MLSS 2015)

Figure 1: From Ghahramani’s slides on GPs and NNs (MLSS 2015)

However, the above does not apply to §neural_networks since they are often so highly parameterized anyway, that they are effectively just as flexible as nonparametric models…

Examples đź”—

Classic Examples of non-parametric analogues to parametric models đź”—

Parametric Non-parametric Application
polynomial regression Gaussian processes function approx.
logistic regression Gaussian process classifiers classification
mixture models, k-means Dirichlet process mixtures clustering
hidden Markov models infinite HMMs time series
factor analysis / pPCA / PMF infinite latent factor models feature discovery

Examples of parametric and nonparametric GANs đź”—

Figure 2: From On How Well Generative Adversarial Networks Learn Densities: Nonparametric and Parametric Results

Figure 2: From On How Well Generative Adversarial Networks Learn Densities: Nonparametric and Parametric Results

Comparison đź”—

Parametric (e.g. GMM) Nonparametric (e.g. kNN, GP)
Strong assumption data distribution Weaker assumption on data distribution.

Parametric models assume the samples are from a specific distribution e.g. from a mixture of Gaussians where the number of Gaussian components is known a priori. This is restrictive since for most real-word problems we cannot know beforehand how complex the data is. For example, a nonparametric method should find the number of Gaussian components itself.

Theoretical equivalence đź”—

Neural networks can also be made nonparametric by taking the number of hidden units to infinity, in which case it turns out that such “infinite hidden layer” neural networks are equivalent to §Gaussian_processes with a certain covariance kernel. See Radford Neal’s 1994 PhD thesis, Bayesian Learning for Neural Networks.