In 1869, the American Missionary Association of New York purchased 500 acres (202 ha) of one of the largest former plantations in central Mississippi to build a college for freedmen and their children, recently freed slaves.

[2] Costs of construction were paid by the United States government through the education department of its Bureau of Refugees and Freedmen.

[3] At this time the school found itself in dire need of expanded facilities and operational funds; an appeal was made by three leaders of Tougaloo University to the Mississippi Superintendent of Public Education for a state role in the institution.

[3] As part of the establishment of the Normal School at Tougaloo, each county in the state was provided with two free scholarships, and every student declaring an intention to teach in Mississippi's common schools was to be allotted a stipend of 50 cents per week out of the state funds for student aid, an amount capped at $1,000 per year.

[5] For the rest of the academic year, classes were conducted in a new barn recently constructed on campus, nicknamed "Ayrshire Hall".

Tougaloo remained predominantly a teacher training school until 1920 when the college ceased receiving aid from the state.

That same year, Sarah Ann Dickey, who had worked with the American Missionary Association since the 1860s, established the Mount Hermon Female Seminary, which would later merge with Tougaloo in 1924,[6] as the two schools had similar ideals and goals.

Built in 1947, the building was named in honor of Fred L. Brownlee, former general secretary of the American Missionary Association.

Included in the African-American portion of the collection are pieces by notable artists Jacob Lawrence, Romare Bearden, David Driskell, Richard Hunt, Elizabeth Catlett and Hale Woodruff.

Begun in 1997 under the leadership of former College trustee, Jane Hearn, the Tougaloo Art Colony affords its participants exposure to and intensive instruction by artists.

Among their holdings are the original papers, photographs and memorabilia of movement leaders including Fannie Lou Hamer, Medgar Evers and Martin Luther King Jr.

The Evers family (trustee Myrlie Evers-Williams and her children with Medgar) donated their home to Tougaloo College for its historical significance.

[23] According to the Tougaloo College website, "over 40% of the African American physicians and dentists practicing in the state of Mississippi, more than one-third of the state's African American attorneys and educators including teachers, principals, school superintendents, college/university faculty and administrators" were trained at the school.

[25] The same magazine reported that in 2022 only 18% of the college's students graduated after four years, placing Tougaloo at the bottom of national rankings.