According to John Stow, it was "The usual place for hanging of pirates and sea-rovers, at the low-water mark, and there to remain till three tides had overflowed them".
[8] Following a fire in the early 19th century, the tavern was rebuilt and renamed The Prospect of Whitby, after a Tyne collier that used to berth next to the pub.
The video for Gilbert O'Sullivan's 1970 hit "Nothing Rhymed" was shot here, as he was living close by in a bedsit when he wrote the song.
In the comicbook The League of Extraordinary Gentlemen Mina Harker pauses in front of the pub and says it brings back memories.
The recurring comic character Hugh Whitby in Harrington's "Capturing the Carlisles" series was named after the pub.
The pub also features in several of the Charles Holborne legal crime thrillers by former barrister, Simon Michael, particularly Corrupted, in which the protagonist and his family work as lightermen on the Thames during the Blitz.