Test oracle

[7] A research literature survey covering 1978 to 2012[6] found several potential categories of test oracles.

Formal specification relies on abstraction, which in turn may naturally have an element of imprecision as all models cannot capture all behavior.

[6]: 518  For example, there may be some implied conclusion from a program crash, i.e. unwanted behavior - an oracle to determine that there may be a problem.

Implicit test oracles may be susceptible to false positives due to environment dependencies.

[6]: 519–520  A quantitative approach aims to find the right amount of information to gather on a SUT (e.g., test results) for a stakeholder to be able to make decisions on fit-for-purpose or the release of the software.

A qualitative approach aims to find the representativeness and suitability of the input test data and context of the output from the SUT.

These can be guided by heuristic approaches, such as gut instincts, rules of thumb, checklist aids, and experience to help tailor the specific combination selected for the SUT.

[15] Documentation that is not a full specification of the product, such as a usage or installation guide, or a record of performance characteristics or minimum machine requirements for the software, would typically be a derived test oracle.

[12]: 466 During Google search, we do not have a complete oracle to verify whether the number of returned results is correct.

A heuristic oracle provides representative or approximate results over a class of test inputs.