It works by bypassing a system security check using a memory exploit (heap overflow) which occurs with USB devices that allows the execution of unsigned code.
The underlying exploit and inner workings of the PlayStation 3 Jailbreak have been entirely reverse engineered[1] and reimplemented through the open source PSGroove.
[2] Through analysis of USB traffic to and from the PS3 Jailbreak, it was found that the device induced a heap overflow in kernel-space memory to trigger execution of untrusted code.
[1] Sony had taken a few steps to prevent the jailbreak of the PlayStation 3, and has associated the action as a form of copyright infringement.
The cases listed below are lawsuits Sony filed in courts to prohibit the sales and imports of circumvention devices that would jailbreak the system.