Strategically positioned at the mouth of the Brisbane River, the hill has been used as a customs lookout, signal and telegraph station, observation post and redoubt commanding the Fort Lytton defence complex, and boys' reformatory.

In August 1857, surveyor James Warner completed a preliminary survey of a site for a village at the south head, which was approved in November 1858, and in December 1858 tenders were called for the construction of a Customs Station nearby on the river.

The Crown and Anchor Hotel at Lytton held a license in 1865-66 - about the time a government wharf adjacent to the Customs Reserve was built in 1866 to tranship railway stores and plant.

[1] Following separation from New South Wales in December 1859, the Queensland Government maintained Lytton's role as the customs entry to the Port of Moreton Bay.

From 1866, Signal Hill, as it became known, also proved a useful post from which to observe semaphore messages from the prison on St Helena Island, which was not connected by telegraph.

[1] Sir George Bowen, on completion of his term as Queensland Governor and departure from Moreton Bay on 4 January 1868, officially named and designated Lytton as Brisbane's port.

The style was the forerunner of the most common 19th century type of post and telegraph office, with decorative finishes to verandah fascias, and sun hoods.

Jervois recommended that the redoubt (an independent fortlet commanding Fort Lytton) be established on Signal Hill, which he considered would be an excellent point whence to watch the movements of an enemy in Moreton Bay.

[1] In July 1878, Scratchley recommended that the occupants of the hulk Proserpine, a former gaol anchored in Moreton Bay which had been refitted c. 1871 as a boys' reformatory, be removed to buildings on Signal Hill as part of the defensible post to be established there.

The redoubt on Signal Hill was to include a large, single-storeyed, hardwood-framed, Reformatory building with chamferboard walls and a shingled roof; kitchen wing; WCs; and a boundary fence enclosing 2 acres (0.81 ha).

To this end the stockade fence on the south side was re-erected 20 yards (18 m) nearer the Reformatory, arrow-headed demi-bastions were formed at the northeast and southwest corners, a ditch was constructed around the fortifications, the trees in front of the redoubt were cleared, a telegraph line was installed from Signal Hill to Fort Lytton below, and ordnance were ordered.

Finally in 1899, just prior to Federation, tenders were called for the removal of the Reformatory buildings from Lytton and their re-erection, with additions, at Westbrook near Toowoomba.

Some improvements to Lytton Hill were made at this time, including the construction a 20-stall timber stables building in 1901–02, and in 1903 the erection of a tent store and barbed-wire entanglement around the Redoubt.

Subsequent construction of the oil refinery and holding tanks has removed most traces of the Second World War defence installation, which included an airfield, with the exception of the top of Lytton Hill and a Second World War anti-aircraft position with concrete bunkers and gun emplacements, in the refinery grounds adjacent to Fort Lytton.

The bulk of these remain, but the former Post and Telegraph Office was vandalised in 1994, resulting in loss of interior casement windows, doors, light fittings and fireplace surrounds and grates.

[1] The principal surviving historical elements in mid-2000 include:[1] Later structures on the site include a sheet and corrugated iron lined garage shed on a concrete slab, located to the southwest of the former telegraph office (010); and SEQEB substation SG1061 - a brick generator shed set on a concrete slab, with a corrugated iron roof and roller door entry (021).

[1] In July 2000 there appeared to be little evidence of the collapsed galvanised iron tank and stand (LH-014) to the southwest of the former Post and Telegraph Office, identified in the 1994 survey; nor of a flagpole (LH-020) comprising a galvanised iron upright with pulley, located northeast of the former cattle grid across the access road on the western side of the hill, and likely associated with a Second World War facility.

A mature Ficus macrophylla (Moreton Bay Fig) to the northeast of the former Post and Telegraph Office may pre-date non-indigenous occupation.

Lytton Hill is important in illustrating the evolution of Queensland's history, being: - a significant communication and observation post at the mouth of the Brisbane River from c. 1860, illustrating the dependence of the Australian colonies on maritime trade and communications in the 19th century; - associated with the early and rapid adoption of the electric telegraph in Queensland in the 1860s/1870s; - a strategic and integral component of the 1880s military facility established at Lytton to defend the Brisbane River; - associated with the conduct of a Boys Reformatory on the Hill from 1880 to 1899; - associated with the activities of the Queensland militia from the early 1880s to the early 1930s and with Queensland preparation for participation in the South African War; - the site of a military hospital during the First World War; - the site of a strategic signals station during the Second World War; - the site of the Control Tower for the Port of Brisbane in the late 20th century.

The place has aesthetic value for its sense of dramatic isolation and ruin within the surrounding well-ordered oil refinery, and for the panoramic views both from and to the hill.