As with the rest of Portland's villages and settlements, Castletown has been designated as a conservation area, as it is a place of special architectural and historic interest.

[1][2] Originally the location for fishermen to launch their boats, Castletown later developed during the 19th century with the construction of Portland Harbour's breakwaters, and the establishment of the naval base.

[7] In 1944 Portland's harbour was commissioned as USNAAB Portland-Weymouth, becoming a major embarkation point for American troops during D-Day.

[3] The navy left in 1995, and Castletown lost the majority of its income, leading to the gradual loss of a number of businesses.

In June 2015 further development plans were revealed for the regeneration of Castletown, including a "scuba diving tourist facility, an American themed D-Day attraction and museum."

[3] To the southern side of the main road of Castletown two large multi-story accommodation buildings sit set into the steeply rising hillside.

Eventually they were acquired by development company Comer Homes, who secured planning consent to convert and refurbish the two buildings to provide a total of 554 1, 2, and 3 bedroom apartments and penthouses.

[16] Castletown D-Day Centre, situated in the old Admiralty Buildings on the dockside, explains the area's role as an embarkation point for the Normandy Lamdings through a collection of wartime artefacts.

[21] Located around 23 metres (75 ft) south of the entrance to The Captain's House is a War Department/Admiralty boundary marker, which has been Grade II Listed since May 1993.