The 1785 house, standing on its original half-acre lot, is in the vernacular "telescopic style" of architecture similar to many Maryland homes, but rare in northern Virginia (see below).

In November 1784, former Lieutenant Colonel (later Major General, Senator, and Governor of Virginia) Henry "Light Horse Harry" Lee III purchased three one-half acre lots in Alexandria from Baldwin Dade, a merchant.

The architect is unknown, but the style is similar to that found at "Hard Bargain", an estate built by the Digges family, and located in Charles County, Maryland, from which Fendall hailed.

The Fendall's, through Elizabeth's first marriage to Phillip Ludwell Lee Sr., had been living at Stratford Hall before moving to Alexandria.

Though continuing to live at Stratford Hall, Light-Horse Harry Lee spent a great deal of time with his relatives at the Fendall home in Alexandria.

Harry Lee was still at the house in April 1789, when George Washington left Virginia to become the first President of the United States.

After Washington's death a large group of citizens met at Lee–Fendall House to make arrangements for Alexandria's participation in his funeral rites.

Matilda's husband Harry Lee wrote to James Madison, Jr.: "You have heard of the loss we have met with in the death of Mrs. Fendall - better for her to be sure had this event taken place sooner & altho' we are convinced of this truth yet our affliction is immoderate.

The multiple tragedies of Harry Lee's failed investments, injuries sustained at the hands of a Baltimore mob as he tried to defend a Federalist friend who had opposed the War of 1812, and his desperate search for health in Barbados, were too much, and Major General Henry Lee III died at Dungeness, Cumberland Island, Georgia on 25 March 1818 while trying to return to his family.

[6] One year after Mollie Fendall's death in 1827, her youngest brother Edmund Jennings Lee I bought the house at auction.

Despite his prominence, Edmund, like his brother Harry previously, encountered financial distress and was forced to first mortgage the house, then sell it at auction in 1833.

Commonly known as "Aunt Turner", she was the granddaughter of Virginia Supreme Court Justice Peter Lyons and confidante of the Cassius Francis Lee, Sr. family.

The young couple were joined at the house by Louis' daughters from his first marriage, Frances and Charlotte Louise, and his father Anthony Charles Cazenove, a French Huguenot immigrant from Geneva.

They added Greek Revival and Italianate embellishments to the original 1785 structure as well as the front and back porches, a third floor addition to the main section, and installed the first heating, plumbing, and servant bell systems in the house.

In 1856, Harriotte moved her small family to her new country home three miles up the road at Seminary Hill (616 Fort Williams Parkway).

When the American Civil War brought invading forces to Harriott Cazenove's door, she fled with her son to her mother's home in Chantilly, Virginia.

The Union Army seized the house for unpaid taxes, but offered to return ownership and pay rent if Harriott Cazenove would swear the Loyalty Oath to the Federal Government.

The suitor was Robert Forsyth Downham, who bought the house for $5,500 thus ending the Lee family's long ownership.

Downham, twice Mayor of Alexandria) wholesale liquor business (whiskey, beer, and wine including the Belle Haven Rye brand).

[4] In 1914, the Downhams hosted President Woodrow Wilson for a reception following that year's George Washington Birthday Parade.

She was responsible for many changes to the house, for example, adding additional glazing to the south porch and painting the exterior white.

Their daughter Katherine, a secretary-treasurer of District 50 of the United Mine Workers at her mother's death, died in 1962 leaving their father the sole family member in the house.

[6] John Llewellyn Lewis[13] started life as a coal miner of Welsh descent in Iowa and quickly worked his way up the union ranks to be president of the United Mine Workers of America (UMWA) for over four decades.

He supported Wendell Willkie for president in 1940 and pulled the miners out on strike in the middle of World War II causing a major energy shortage.

During the strike, Lewis was strung up in effigy at the corner of Washington and Oronoco, outside Lee-Fendall, as a traitor for hurting the war effort so much.

The son then sold it to the Virginia Trust for Historic Preservation, a 501(c)(3) non-profit foundation doing-business-as Lee–Fendall House Museum and Garden.

The Trust was created in 1968 by Jay W. Johns with encouragement from Frances Shively in hopes of purchasing and preserving the house upon Lewis' death.

The Trust received financial assistance from the Commonwealth of Virginia and the City of Alexandria in completing funding for the purchase.

Memberships, donations, small grants, and revenue gathered from tours, events, and a gift shop were insufficient to meet the Museum's needs.

First Lady Betty Ford personally helped raise funds for the restoration and it was designated Alexandria's official Bi-centennial Garden.