The series stars Hope Lange as Carolyn Muir, a young widow and writer who rents Gull Cottage, near the fictional fishing village of Schooner Bay, Maine, and moves into the rental with her two children, a housekeeper (played by Reta Shaw), and their dog.

The cottage is haunted by the ghost of its former owner Daniel Gregg, a 19th-century sea captain, who died in 1869, played by Edward Mulhare.

Charles Nelson Reilly plays Claymore Gregg, the great-nephew of the captain, who rents the cottage to Mrs. Muir without telling her it is haunted by his ancestor.

During its first year on NBC, it followed Get Smart; its Big Three competition was My Three Sons on CBS and the first half of The Lawrence Welk Show on ABC.

The following year, Family Affair was cancelled by CBS after a five-year run because it was in the same time slot as the NBC hit variety series The Flip Wilson Show.