Sharpe commissioned the design and construction of a pavilion, gardens, parks, and entrance court of this 1,000-acre (400 ha) estate and a house for his intended bride, Mary Ogle.

Work continued to add wings on either side until 1769, when it became Sharpe's residence following his unexpected relief as governor after 16 years by Robert Eden, a relative of Lord Baltimore.

According to a 1912 biography by Matilda Ridout Edgar, Sharpe "spent as much of his time as was possible at Whitehall, amusing himself with his favourite pursuit of farming... His Garden was his passion, and seeds and scions of trees and rare shrubs and flowers to beautify it were sent for from Holland and England and France."

Although Whitehall is not considered a plantation as it did not specialize in commercial sale of cash crops, It is noted that most of the labor on the farm and gardens was provided by "his large retinue of negro slaves and indentured white servants.

While the home was built originally by provincial Governor Sharpe as a retreat and entertainment pavilion; it was later enlarged and became his residence from his retirement in 1769 until his return to England in 1773.

Whitehall is an unusually long one-story five-part Flemish bond brick building, about 200 feet (61 m) in length, with a two-story elevation on the north side.

On the north side the ground falls away, revealing a full basement story beneath the main house, with additional extensions running beyond the end pavilions.

The south portico faces the Severn River, while a small balcony on the north side of the main hall leads to a pair of stairs to grade level.

The second story was removed during the comprehensive restoration that began in the 1950s under Charles Scarlett, Jr.[3] The site immediately to the north of the house is surrounded by earth mounds in the form of a bastioned breastwork.