Fort Phillip was a citadel style fort constructed as part of the first defensive works of the penal settlement of Sydney, located on Observatory Hill in the Sydney suburb of Millers Point, New South Wales, Australia.

Governor John Hunter ordered a fort to be built and Fort Phillip was commissioned in 1804 by the third New South Wales Governor (in office 1800–1806), Philip Gidley King, partly as a response to external threats such as a possible French attack in 1803 and partly due to the internal unrest from rebellious convicts reflected in Australia's only major convict rebellion at Castle Hill in March 1804.

Windmill Hill (later Observatory Hill) was chosen as a fort location as it was the highest point above the colony, affording commanding views of the Harbour approaches from east and west, the river and road to Parramatta, surrounding country and of the entire town below.

[1][2] During 1847-48, a new signal station, designed by the colonial architect Mortimer Lewis, was built to replace the flagstaff.

[3] Sydney Observatory was built on the site in 1858 to provide accurate time readings for the colony and to observe stars in the southern sky.