Buffer overflow protection is any of various techniques used during software development to enhance the security of executable programs by detecting buffer overflows on stack-allocated variables, and preventing them from causing program misbehavior or from becoming serious security vulnerabilities.
This almost always results in corruption of adjacent data on the stack, which could lead to program crashes, incorrect operation, or security issues.
By verifying the canary value, execution of the affected program can be terminated, preventing it from misbehaving or from allowing an attacker to take control over it.
This almost always results in corruption of adjacent data on the stack, and in cases where the overflow was triggered by mistake, will often cause the program to crash or operate incorrectly.
If the affected program is running with special privileges, or if it accepts data from untrusted network hosts (for example, a public webserver), then the bug is a potential security vulnerability that allows an attacker to inject executable code into the running program and take control of the process.
The terminology is a reference to the historic practice of using canaries in coal mines, since they would be affected by toxic gases earlier than the miners, thus providing a biological warning system.
This prevents attacks using strcpy() and other methods that return upon copying a null character, while the undesirable result is that the canary is known.
[13] It improved on the idea of StackGuard by placing buffers after local pointers and function arguments in the stack frame.
Red Hat engineers identified problems with ProPolice though, and in 2005 re-implemented stack-smashing protection for inclusion in GCC 4.1.
[25] Stack protection is standard in certain operating systems, including OpenBSD,[26] Hardened Gentoo[27] and DragonFly BSD.
[31] Clang supports the same -fstack-protector options as GCC[32] and a stronger "safe stack" (-fsanitize=safe-stack) system with similarly low performance impact.
[33] Clang also has three buffer overflow detectors, namely AddressSanitizer (-fsanitize=address),[6] UBSan (-fsanitize=bounds),[34] and the unofficial SafeCode (last updated for LLVM 3.0).
[35] These systems have different tradeoffs in terms of performance penalty, memory overhead, and classes of detected bugs.
[36] Intel's C and C++ compiler supports stack-smashing protection with options similar to those provided by GCC and Microsoft Visual Studio.
[37] Fail-Safe C[7] is an open-source memory-safe ANSI C compiler that performs bounds checking based on fat pointers and object-oriented memory access.
[38] Invented by Mike Frantzen, StackGhost is a simple tweak to the register window spill/fill routines which makes buffer overflows much more difficult to exploit.
It uses a unique hardware feature of the Sun Microsystems SPARC architecture (that being: deferred on-stack in-frame register window spill/fill) to detect modifications of return pointers (a common way for an exploit to hijack execution paths) transparently, automatically protecting all applications without requiring binary or source modifications.
The resulting gdb issues were resolved by Mark Kettenis two years later, allowing enabling of the feature.