First known as Oldtown, it was renamed in 1870 for the prominent Goldsborough family who owned land in the area.

Castle Hall was listed on the National Register of Historic Places in 1975.

Three state highways serve the town: Maryland Route 313, which, while signed north-south, comes in from the south but turns toward the west within the town; Maryland Route 311, which heads north out of town from MD 313; and Maryland Route 287, which departs eastward from MD 313 in Goldsboro.

19.5% of all households were made up of individuals, and 12.6% had someone living alone who was 65 years of age or older.

The racial makeup of the town was 93.98% White, 5.56% African American, and 0.46% from two or more races.

20.8% of all households were made up of individuals, and 7.8% had someone living alone who was 65 years of age or older.

About 5.1% of families and 9.6% of the population were below the poverty line, including 21.7% of those under the age of eighteen and none of those 65 or over.