The Woolhope Club was founded in 1851 "for the practical study, in all its branches, of the Natural History of Herefordshire and the districts immediately adjacent".

In 1869 the club's president, Sir James Rankin MP, offered money for the building of a library and museum during his presidential retirement speech of 22 February 1870.

The resolution was passed and Rankin provided the capital needed to purchase a strip of land on Broad Street in the centre of the city for £1,750 from Mr William Beavon of The Villa, Whitecross Road.

[11] Due to structural issues and fire regulations only ten people were allowed into the museum and art gallery at anyone time from 2020 after the COVID-19 lockdowns.

Alongside the Woolhope Club, members of other local natural history, literary, philosophical and antiquarian societies generously donated to the museum.

In 1927 a public appeal raised funding to mount two Roman floor mosaics from Kenchester in the stair hall of the building.

[15] The museum displays were left largely unchanged from 1960 until the turn of the century in expectation the building would be refurbished and new home found for the library.

In 2005, the museum became the first in the United Kingdom to invest in the Talking Tactile Tablet (T3),[17] developed at the Royal National College for the Blind together with a software company based in the USA.

On closing for refurbishment in 2023 exhibits in the museum included a two-headed calf, a two-metre long fish, various swords, elements of costume and textiles, as well as objects of historic, social and scientific interest dating back from the pre-historic era up to the 20th century.

An area of the art gallery was set aside for the permanent display of works by Brian Hatton, as some of his materials and selected letters.

A regular programme of temporary exhibitions of craftwork, paintings, photography, and prints was shown in the art gallery.

[22] The art gallery hosted Grayson Perry's exhibition, The Vanity of Small Differences, from October to December 2021.