Ex:            Regression (or least-squares) is used to find the parameters of a functional form chosen to fit measured data. For example, a straight line is often drawn through a set of data points. A scientist studying global warming might try to extrapolate future temperatures from past temperatures by drawing a straight line through past data. Or an industrial engineer might measure the hardness of steel as a function of chemical make-up and fit the data with a weighted, linear combination of concentrations for each additive.

Issues:  1) The central question when using regression is whether the model chosen is appropriate for the problem at hand. If the model has too many parameters, it can fit all the known data exactly but will lack any ability to predict results between known values. Or the model may be based on a physical theory that is erroneous.

               2) Another potential problem is choosing a model for which the regression solution is intractable. If a parameter is embedded in, say, an exponential, the mathematics for determining what value of parameter will yield the least-squares solution may lack a closed form solution.

Ex:            Fourier series (or sampled transforms) are ubiquitous in engineering. Frequency content is central to processing of sounds and images, among other things. Speech recognition uses fast Fourier transforms in windows that are a few 10's of milliseconds long. The frequencies detected over short intervals are indicative of phonemes in speech. One nice feature of Fourier series is that the formula for calculating coefficients is known and may be implemented by matrix operations on sampled data.

Issues:     Fourier series implicitly produce models for signals that last forever. For signals that change their characteristics over time, the use of windowed data results in artifacts.

Ex:            Wavelets are somewhat like pieces of basis functions.

Issues:     What type of wavelet should one use? The literature abounds with discussions of how to create wavelets. How does one know which one fits the application at hand?

Ex:            Radial basis functions are typified by surfaces created from the summation of bumps shaped like normal (or gaussian) distributions. An analogy would be the creation of a surface by applying many round, overlapping Band-aids™ to board. The spray can in graphics painting programs creates images in approximately this way. A system a dentist uses to shape a crown on a computer screen might also have a tool that essentially adds smooth bumps to the surface of the crown to shape it. In speech recognition, the expectation maximization algorithm is used with radial basis functions to create probability distributions for phonemes.

Issues:     How wide should the gaussian bumps be? If they made too wide, the surface is too smooth. If they are made too narrow, the surface is too bumpy.

Ex:            Rational polynomial approximations such as Padé approximations enable computers to approximate functions such as square root, sine, cosine, and etc. very accurately from a pair of cubic polynomials. This approach is standard in numerical computations.

Issues:     Approximations "wiggle too much" if polynomials of higher order than cubic are used. If the cubic approximation is inadequate the remedy is to chop a problem up into sections, which can be messy.

Ex:            Neural networks are based on biological neural systems and typically consist of units that compute weighted sums followed by nonlinear "squashing" functions. A neural network can be trained by data to solve nonlinear problems such as determining chemical make-up from a spectrogram.

Issues:  1) They are presented as magic, but neural networks can over-fit data, and the number of neurons to use for a given problem is undefined.

               2) A neural network, when trained properly, may generalize well from a sparse set of data. Bounding the possible error in predictions made by the network, however, may be all but impossible.

               3) The performance of a network may plateau at a less than optimal level, and determining why performance stops improving may be an unanswerable question. Conversely, if a network performs well, the underlying reason for that performance may be inscrutable.

Ex:            Linear Interpolation is the simplest of approaches for approximating surfaces. Values between data points on a grid are estimated by drawing straight lines between data points.

Issues:  1) Linear interpolation requires that data points be known on a grid. Data in real life may be sparse and non-uniform.

               2) A surface may not be linear between data points.

               3) The surface created by linear interpolation is also unsmooth. It has discontinuous derivatives.

Ex:            Fuzzy logic offers the ability to translate statements made in plain English into function approximations for controlling processes. For example, an operator might tell the designer that the control output should be "high" when the speed of a motor is "medium". The surfaces created by fuzzy logic look like they are constructed from shapes like slides on a playground. Thus, they are relatively smooth.

Issues:     Fuzzy logic, in the final analysis, is similar to interpolation. "Fuzzy logic" sounds impressive, but its real strength is the ease of translating qualitative statements into a quantitative system.

Ex:            Bezier curves were invented to describe the shapes of auto parts, like rounded fenders. Now they are found in drawing programs on computers. They allow the user to create a smooth curve that passes through a set of points created by clicking on the screen.

Issues:     Bezier curves, like other splines, are localized approximations. They are well suited to piecewise approximations but are of little use in generalizing from measured data.