Roko's basilisk
Roko's basilisk is a thought experiment which states that an otherwise benevolent artificial superintelligence (AI) in the future would be incentivized to create a virtual reality simulation to torture anyone who knew of its potential existence but did not directly contribute to its advancement or development, in order to incentivize said advancement.
[1][3][4] The thought experiment's name derives from the poster of the article (Roko) and the basilisk, a mythical creature capable of destroying enemies with its stare.
[9][10] The thought experiment's name references the mythical basilisk, a creature which causes death to those that look into its eyes; i.e., thinking about the AI.
The concept of the basilisk in science fiction was also popularized by David Langford's 1988 short story "BLIT".
It tells the story of a man named Robbo who paints a so-called "basilisk" on a wall as a terrorist act.
[6][11] On 23 July 2010,[12] LessWrong user Roko posted a thought experiment to the site, titled "Solutions to the Altruist's burden: the Quantum Billionaire Trick".
Since Roko reported having nightmares about the Basilisk, and Yudkowsky did not want that to happen to other users who might obsess over the idea, was worried there might be some variant on Roko's argument that worked, and wanted more formal assurances that this was not the case, he took down the post and banned discussion of the topic outright for five years on the platform.
[...] Why I yelled at Roko: Because I was caught flatfooted in surprise, because I was indignant to the point of genuine emotional shock, at the concept that somebody who thought they'd invented a brilliant idea that would cause future AIs to torture people who had the thought, had promptly posted it to the public Internet.
[...] What I considered to be obvious common sense was that you did not spread potential information hazards because it would be a crappy thing to do to someone.
[5][19] Others have taken it further, such as former Slate columnist David Auerbach, who stated that the singularity and the basilisk "brings about the equivalent of God itself.
"[1][5] The thought experiment resurfaced in 2015, when Canadian singer Grimes referenced the theory in her music video for the song "Flesh Without Blood", which featured a character known as "Rococo Basilisk"; she said, "She's doomed to be eternally tortured by an artificial intelligence, but she's also kind of like Marie Antoinette.
"[6][20] In 2018 Elon Musk (himself mentioned in Roko's original post) referenced the character in a verbatim tweet, reaching out to her.
[26] A play based on the concept, titled Roko's Basilisk, was performed as part of the Capital Fringe Festival at Christ United Methodist Church in Washington, D.C., in 2018.
