The area was largely destroyed by the Luftwaffe during the Blitz, and that which remained was subsequently demolished by 1969, the last demolitions being the Bear & Ragged Staff and the Cat and Wheel pubs in Little Peter Street.

The ruined 14th-century tower of St Mary-le-Port Church stands to the west of the park, surrounded by derelict financial office buildings.

[3] Archaeological excavations of the area in 1962-1963 suggest that the original settlement was centred to the east of St Peter's Church, with Mary-le-Port on its western limit.

Bristol Castle was home to the Norman garrison, and was a massive structure whose keep was comparable to that of the Tower of London.

[6][7] By the 16th century the castle was 'tending to ruin', and following his victory in the English Civil War Oliver Cromwell decreed its destruction in 1655, a process which was completed within a fortnight.

[11] The area also hosted a cinema (the News Theatre) and its narrow winding back-streets contained many independent shops, hotels, and pubs.

The idea that Bristol's main shopping area should be moved away from the Castle Street and Wine Street area was first proposed to Bristol Corporation's Planning and Reconstruction Committee by the Multiple Traders' Federation (MTF), representing the larger retailers and chain-stores, in October 1943.

This suggestion formed part of the City Engineer's Master Plan of February 1944, which envisaged that the area would contain a limited number of buildings such as a conference hall, with underground parking for 2,000 cars.

[22] The main objection to this plan came from the independent traders who did not wish to relocate because they felt that the proposed new shopping area was 'off the beaten track'.

[22] However, when the Bristol Retail Traders Federation, representing the small traders, produced a poll showing that 13,000 people wished to retain the shopping centre on its current site whereas only 400 preferred to see it moved, the local Labour party dismissed the poll describing it as 'undemocratic' and comparing the Federation's methods to those of Hitler.

[1] In the event, the north-western corner was leased to the Bank of England and Norwich Union Insurance (whose offices now stand derelict), and the critical corner site where the Dutch House had stood was replaced by a link road to a short section of dual-carriageway linking High Street with Wine Street.

In 2006, Bristol City Council announced its intention to redevelop the area to the west of Castle Park, known as the 'Mary-le-Port' site.

These plans encompassed the by then derelict financial buildings, and the area to the west of St Peter's Church, about 5% of the park.

[26] It was to be a 400,000 square feet (37,000 m2) mixed-use development to 'regenerate this historic heart of Bristol'[27] and to 'improve connections between Broadmead, the Old City and Redcliffe' and 'enhance the setting of St Mary-le-Port church'.

[30] In March 2016, Bristol City Council approved plans for an S-shaped footbridge over the Floating Harbour, linking the park to the Finzels Reach development.

[37] The ruins of St Peter's Church are a memorial to the civilians and auxiliary personnel killed in the aerial bombing of Bristol,[38] and a plaque on its wall lists their names.