Look-elsewhere effect

[1][2][3][4][5] Once the possibility of look-elsewhere error in an analysis is acknowledged, it can be compensated for by careful application of standard mathematical techniques.

[6][7][8] More generally known in statistics as the problem of multiple comparisons, the term gained some media attention in 2011, in the context of the search for the Higgs boson at the Large Hadron Collider.

[9] Many statistical tests deliver a p-value, the probability that a given result could be obtained by chance, assuming the hypothesis one seeks to prove is in fact false.

In order to compensate for this, you could divide your threshold α by the number of tests n, so a result is significant when p < α/n.

Or, equivalently, multiply the observed p value by the number of tests (significant when np < α).