The colonial estate was built by his father, John Tayloe II, on the north bank of the Rappahannock River across from Tappahannock, Virginia.

He was educated at Eton and Christ's College, Cambridge University in England, served in the Virginia state legislature, and ran unsuccessfully for Congress in 1800.

[3] The Octagon was originally constructed to be a winter residence for the Tayloe family, but they lived in the house year-round from 1818 to 1855.

The Tayloes were involved in shipbuilding, horse breeding and racing, and owned several iron foundries—they were fairly diversified for a plantation family.

[6] On April 19, 1797, Tayloe paid $1,000 (~$22,685 in 2023) to Gustavus W. Scott for lot 8 in Square 170, at the corner of New York Avenue and 18th Street NW, as laid out in a plan of the city by Pierre Charles L'Enfant and surveyed by Andrew Ellicott.

On April 19, 1799, Dr. William Thornton wrote to George Washington, "Mr. J. Tayloe, of Virginia, has contracted to build a house in the City near the President's Square of $13,000 value."

His first problem was to plan a house that would fit the lot, the south side of which was cut away on the bias by the diagonal of New York Avenue.

The kitchen, stable and outhouses are built of brick and accommodated a large number of both servants and horses.

All the work in the circular vestibule coincides with the circumference of the tower, the doors, sash and glass being made on the circle.

The oldest son, John Tayloe IV, served in the US Navy during the War of 1812 aboard the USS Constitution.

John Tayloe III was a Federalist, and not terribly supportive of President James Madison and the war with England that began in 1812, but he was active in the Virginia militia and commanded a regiment of DC cavalry.

[7] The house probably would have been spared even if it hadn't been effectively a "diplomatic residence", since the British were under strict orders not to damage private property.

When First Lady Dolley Madison fled the city as the British approached, she sent her pet parrot to the French consulate at the Octagon for safekeeping.

President James Madison and his wife, Dolley moved into the Octagon on September 8, 1814, after the burning of the White House by British forces.

President Madison ratified the Treaty of Ghent, which ended the War of 1812, in the upstairs study at the Octagon on February 17, 1815.

[8][9] While a resident of Washington, Tayloe helped found and organize St. John's Episcopal Church, Lafayette Square in 1814, served as a trustee in 1816 during its construction and upon completion served on the vestry and donated to the parish a communion service of silver, which Bishop William Meade, in his work on the old Churches of Virginia, says had been purchased by Col. Tayloe at a sale of the effects of the Lunenburg Parish Church in Richmond County, VA., to prevent its desecration for secular use.

[10] "Another invitation recalls one of General Washington's closest friends, whom he persuaded to become a resident of Washington in its infancy, and who built the spacious mansion on the corner of New York Ave and Eighteenth Street, which is one of the surviving relics of the primitive city, not having been destroyed by the British in 1814 - Col. Tayloe: "Mr. Tayloe requests the favor of Mr. Van Rensselaer to dine with him on Sat next at 4 o'clock.

[2][12] The museum is currently owned and operated by the Architects Foundation and is open to the public Friday and Saturday 11am-4pm.

The Octagon achieves a zenith in Federal architecture in the United States, through a plan which combines a circle, two rectangles, and a triangle, and through the elegance and restraint of the interior and exterior decoration.

The construction materials, such as bricks, timber, iron, and Aquia Creek sandstone were all manufactured locally.

[16] There are many variations of ghost stories that feature the Octagon House in Washington, D.C. As one of the oldest buildings in the city, it lends itself nicely as a setting for historical, spooky, and macabre tales.

The stories recorded here are merely a presentation of a few of the reported experiences and legends that have evolved over the past 200 years, and should not be taken as historical fact.

"[3] An account by Marian Gouverneur, wife of Samuel Laurence Gouverneur Jr., the first American consul in Foo Chow in China, said that the General George D. Ramsay, Chief of Ordnance for the United States Army and commander of the Washington Arsenal in Washington, D.C., and his experience with the bells: "I have been told by the daughters of General George D. Ramsay that upon one occasion their father was requested by Colonel John Tayloe...to remain at the Octagon overnight, when we was obliged to be absent, as a protection to his daughters… While the members of the family were at the evening meal, the bells in the house began to ring violently.

Colonel Tayloe and his daughter quarreled on the second floor landing over the girl's relationship with a British officer stationed in the city.

Her specter is allegedly seen crumpled at the bottom of the steps or on the stairs near the second floor landing, and sometimes exhibits itself as the light of a candle moving up the staircase.

From all quarters the party rushed… Too brave to desert, yet cowards at heart, we watched the gray light of morning dawn, and each man of us thanked God his night among ghosts was past.

After those screams our band was closely knit together… collectively we listened through the waning hours of night to the clanking of sabers and tramping of footfalls.

"[25] Museum superintendent Alric H. Clay claimed that in the 1960s spirits would often turn on the lights and open The Octagon's doors late at night.

[27] A gambler shot to death in the home's third-floor bedroom in the late 19th century has been reportedly seen in the room in which he died.