The unwarranted confidence which is produced by a good fit between the predicted outcome and the input information may be called the illusion of validity.
Candidates were taken to an obstacle field and assigned a group task so that Kahneman and his fellow evaluators could discern their individual leadership qualities or lack thereof.
But although Kahneman and his colleagues emerged from the exercise with very clear judgments as to who was and wasn't a potential leader, their forecasts proved "largely useless" in the long term.
Decades later, Kahneman reflected that at least part of the reason for his and his colleagues' failure in assessing the officer candidates was that they had been confronted with a difficult question but had instead, without realizing it, answered an easier one instead.
[4] The scientist Freeman Dyson has recalled his experience as a British Army statistician during World War II, performing an analysis of the operations of the Bomber Command.
At the time, an officer argued that because of the heavy gun turrets they carried, the bombers were too slow and could not fly high enough to avoid being shot down.
He was not alone: everyone in the command "saw every bomber crew as a tightly knit team of seven, with the gunners playing an essential role defending their comrades against fighter attack."
[8] Other cases where this phenomenon appears include job interviews, wine tasting, prediction of prices in the financial international markets, political strategy.
Among the factors contributing to the illusion of validity, according to Meinolf Dierkes, Ariane Berthoin Antal, John Child, and Ikujiro Nonaka, are "a person's tendency to register the frequency of events more than their probability"; "the impossibility of gathering information about alternative assumptions if action is based on a hypothesis"; a "disregard of base-rate information"; and "the self-fulfilling prophecy," or "a behavior manifested in individuals or groups because it was expected.
"[12] If one wishes to try to avoid the illusion of validity, according to Kahneman, one should ask two questions: Is the environment in which the judgment is made sufficiently regular to enable predictions from the available evidence?
"[4] In his article on the false rape case at the University of Virginia, Harlan Loeb outlined an approach to avoiding the illusion of validity in cases which, like that one, involve “a highly emotional and personal issue that has national resonance, high-profile media coverage and an organization already on the defensive with recent issues.” He advised, first: "Always challenge (appropriately, of course) facts and assumptions that many rely on to inform their thinking and decision-making about risk and crisis management."
Second: "In situations with palpable unknowns, where the illusion of validity in decision-making is a material threat, push hard to do research, polling and active listening to help identify the levers and pulleys that shape the operating and environmental realities of the present risk."
Third: "Determine existing organizational challenges that will prevent leadership from making decisions consistently, effectively, and quickly in the face of uncertainty."