Bayesian optimization

Bayesian optimization is a sequential design strategy for global optimization of black-box functions, that does not assume any functional forms. It is usually employed to optimize expensive-to-evaluate functions. With the rise of artificial intelligence innovation in the 21st century, Bayesian optimization algorithms have found prominent use in machine learning problems for optimizing hyperparameter values.

History

The term is generally attributed to Jonas Mockus and is coined in his work from a series of publications on global optimization in the 1970s and 1980s.

Early mathematics foundations

From 1960s to 1980s

The earliest idea of Bayesian optimization sprang in 1964, from a paper by American applied mathematician Harold J. Kushner, “A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise”. Although not directly proposing Bayesian optimization, in this paper, he first proposed a new method of locating the maximum point of an arbitrary multipeak curve in a noisy environment. This

…

Source: Wikipedia