The building now known as Saybrook and Branford Colleges was built as the Memorial Quadrangle on the site of what was once the old gymnasium.

[2] The courtyards are named for the towns Yale occupied before its move to New Haven: Killingworth Court after Killingworth, Connecticut, where Rector Abraham Pierson first held classes, and Saybrook Court after Old Saybrook, Connecticut, where it resided as the Collegiate School from 1703 to 1718.

Around the entryways are the stone heads of various associates of Yale University, including Vance McCormick, former chairman of the Yale Corporation's architectural planning committee, and Russell Chittenden, former director of the Sheffield Scientific School.

In Saybrook Court are the arms of several American universities and of Elihu Yale and Edward Harkness.

[3] The arms of Saybrook College are the quartering of the arms of William Fiennes, 1st Viscount Saye and Sele and of Robert Greville, 2nd Baron Brooke, who were the early promoters of the Saybrook Colony, where Yale would later be founded.

Male and female college residents strip down to their underwear (some seniors remove all their clothing during The Game[6]) The words to the Saybrook strip song change to accommodate the names of the current Head of College and Dean.

[8][9] Mary Miller, a scholar of Mesoamerican art, was appointed master in 1999 to restore the college's structure and morale.

In the fall of 2009, computer science professor Paul Hudak became the ninth master of Saybrook.