At an executive meeting on October 2, 1903, school President Francis Preston Venable announced that former North Carolina Supreme Court justice William Preston Bynum donated $25,000 to have a gymnasium built in honor of his grandson who was a student at the university and had died due to typhoid fever.

It originally contained a swimming pool, gymnasium, office spaces, and other rooms for various sports like boxing and fencing.

The gym was placed into control of Dr. Robert Lawson, a skilled gymnast and the former coach of the school's baseball team.

After the basketball team's departure, Bynum was still used by students for activities; however, the pool was closed in 1924 due to having an inadequate filtration system.

On October 2, 1903, during an executive university committee meeting on hosted in Governor Daniel Lindsay Russell's office, school President Francis Preston Venable revealed former North Carolina Supreme Court justice William Preston Bynum donated $25,000 to the school to build a gymnasium.

[6] At the time of the plans' announcement, the location of the future gym was reported to be on top of or near an "old athletic field" near the library.

[8] In February 1904, President Venable announced that the building's plans had been approved by Bynum and the trustees and work would begin "at once", weather permitting.

[8] The building itself was created with a Greek influence and was to feature grey mottled brick and terra cotta trimmings.

[10] The prospect of building the new gym caused some relief with regards to first Memorial Hall, which had been hosting physical activities and suffered many broken windows.

[11] With your permission, I have caused to be erected on the grounds of the university, a gymnasium, intended for the use and benefit of the students, and in memory of a grandson who died before his graduation at this school.

The Morning Post reported on May 10, 1904, that material for the gymnasium had arrived to campus and work was expected to be starting that week.

[17] An update on construction in November was published by The Evening Tribune, where it stated that the building was coming along rapidly, but the new target finish date was in February 1905.

[26] Dr. Richard H. Lewis officially accepted the building for the university and commented: "Nothing appeals more to lusty youth than athletics in all its forms.

[28] While Bynum Gymnasium had primarily been used for student's gymnastics beforehand, in 1910 an author for The Tar Heel wrote that it could be used as the site for the basketball team to practice because the building was not in use at night.

[30] The first basketball game in Bynum was held on January 27, 1911, when North Carolina hosted Virginia Christian and won 42–21.

[31] Playing basketball in Bynum was known to have many issues, including the floors being slick, the backboards being "a good deal liver" than most, and lighting being poor.

[30] The pool was closed in 1924 because of an inadequate filtration system and was subsequently used as space for Playmakers Theatre until the building was repurposed.

[23] The final intercollegiate basketball game in Bynum was held on February 24, 1923, when the Tar Heels faced Virginia and won 39–16.

[38] The Tar Heel wrote that with the men's basketball team's relocation to the Tin Can, it would not be as highly trafficked, thus allowing for more physical education courses.

[5][41] During World War II, the university radio station shut down its operations in Caldwell Hall in June 1942.