Random search

Anderson in 1953 reviewed the progress of methods in finding maximum or minimum of problems using a series of guesses distributed with a certain order or pattern in the parameter searching space, e.g. a confounded design with exponentially distributed spacings/steps.

The method was developed to screen the experimental conditions in chemical reactions by a number of scientists listed in Anderson's paper.

A MATLAB code reproducing the sequential procedure for the general non-linear regression of an example mathematical model can be found here (JCFit @ GitHub).

[2] The name "random search" is attributed to Rastrigin[3] who made an early presentation of RS along with basic mathematical analysis.

A number of RS variants have been introduced in the literature with structured sampling in the searching space:

Scheme of random search using a non-linear regression problem as an example. The goal is to minimize the value of the penalty function. The right bottom shows a few example methods: 1. Non-structured random search, 2. structured random search, 3. Gauss-Newton algorithm , and 4. Levenberg-Marquardt algorithm . 1,2 do not need to know the gradient and 3,4 have to calculate the gradient and usually minimize on both A and k parameters at the same time (scheme only shows the k dimension).