The building houses the facilities of many varsity teams at Yale, including basketball, fencing, gymnastics, squash, swimming, and volleyball.

[2] For the design of Payne Whitney Gymnasium, architect John Russell Pope was awarded the Silver Medal at the 1932 Olympic Games Art Competition.

The wing opposite the Amphiteater houses the Robert J. H. Kiphuth Exhibition Pool (6 lanes, 25 yards), where the swimming teams compete.

[3] The tower itself contains the Kiphuth Trophy Room (where mementos from Harvard-Yale game balls to Olympic gold medals are displayed), several multi-purpose recreational areas, the fencing salon, and the gymnastics studio.

The Lanman Center, located behind the Amphitheater wing, provides a vast spread of additional flexible floor space, with a balcony running track ringing the facility.