A classic four-over-four Greek Revival home, one of six of this type in Madison, it features a wide front porch supported by four large scamozi fluted columns, eight 20' × 20' rooms plus three additional ones in back, 12½ foot ceilings downstairs, and nine fireplaces upstairs and down.

The two-story brick structure was established by charter of the Georgia Legislature on December 16, 1815, and supported in part by the state in the form of fines and forfeitures levied in criminal prosecutions.

According to Samuel Burney's letters home mentioned below, the Carter Shepherds were one of the most prosperous families in Morgan County, with a 2,440-acre (9.9 km2) plantation to the south of Madison and a small sawmill.

On November 8, 1860, Mrs. Shepherd's eldest daughter, Sallie, then 20, married Samuel A. Burney (4/26/1840-3/22/1896), who was the same age, a native of Morgan County and an honor graduate of Mercer College in Macon, Georgia.

In September 1861, after teaching a year, he joined the Panola Guards of Thomas Cobb’s Georgia Legion, composed of many men from Madison and the county, and fought for the remainder of the war, mostly in Virginia.

Another soldier at the Shepherd House on the morning of November 18 was Captain Charles W. Baldwin, one of Morgan County’s most celebrated Civil War heroes.

Having been wounded in the elbow at the Battle of Atlanta earlier that year and still not recovered, he was visiting Mrs. Shepherd's second daughter, Annie, on the eve of Madison's most momentous day.

Though barely able to ride horseback, my arm still in a sling, I undertook the trip of refugee…(and was captured two days later by Union troops, ending up in a horrid prison on Hilton Head Island, South Carolina and, later, a tolerable one in Washington, DC).”[4]In April 1865, at the end of the war, Sam Burney came back to Madison to live at the Shepherd House with his wife Sallie and run his mother-in-law’s plantation.

Three months later, Captain Baldwin was released from prison and returned “wearing a splendid suit of clothes, which I had bought in Philadelphia with the money I had gotten for the Confederate notes I had sold”.

Mary had lost her husband Stuart Anderson in the war in 1864, was without children, and found herself unable to maintain the Joseph Watson plantation house along the Oconee River.

Mary, the second owner of the Shepherd House, brought only several of the finest pieces of Joseph Watson family furniture, also chairs and china, the remainder being sold at an auction at her death in 1880.

Miss Josie traced her Watson relatives of Green County back to the mid-18th century in Virginia, her triple-great grandfather being Charles Bonar.

After the Civil War, Edward Payson had left his large plantation between Brownwood and Little Creek, 5+ miles south of Madison, for downtown Social Circle so that the children could attend school but he held on to the land.

The changes included, most notably, widening the porch to its present size and replacing the capitals and bases of the simple Doric columns with the fancier Ionic ones of today.

A modest man, highly regarded in the community and a close relative of the first Chief Justice of the Georgia Supreme Court, he was a church deacon, on the Board of Directors of the First National Bank of Madison for many years, and a member of various civic organizations.

In 1968, he was elected to the Georgia Newspaper Hall of Fame at UGA's Grady College of Journalism,[7] and his portrait-like photograph hangs next to that of Ralph McGill, his political opposite.

The youngest of five children, Polly had attended William & Mary College and had been a part-time dancer and worked in San Francisco and in Washington at the Egyptian Embassy before meeting Edward.

He not only opened a patent office in downtown Atlanta, but also started managing the family-owned Madisonian newspaper and the unprofitable family farm and peach shed, which were later sold.

Polly Newton headed up the Morgan County Chapter of the Red Cross for well over a decade and was one of the key Madisonians to re-establish the local Episcopal Church, she conducting the first confirmation classes.

After his death from poor health on January 3, 1983, at age 78, Mrs. Newton continued opening the house during the city's periodic tours, traveling overseas, and hosting Weihe-Newton family reunions.

In 1991, a scene from the made-for-TV movie, The Perfect Tribute, about the events leading up to the Gettysburg Address, was shot in the area back of her home, and she forever raved about how easy it was talking to the affable Oscar-winning star, Jason Robards.

She died of Alzheimer's disease on October 12, 2003, leaving the house to her three sons, and is buried alongside her husband in the Newton plot at the front of Old Madison Cemetery.