Built from 1927 to 1938, it is named after Charles Fuller Baker, the second and longest-serving dean, University of the Philippines Los Baños College of Agriculture.

[2] [3] It was part of the Los Baños Internment Camp for American and allied POWs and civilians during the Japanese Occupation of the Philippines during World War II.

[5] During World War II, Japanese soldiers occupied Los Baños including the UPLB Campus.

[2] To rescue the prisoners, a raid on the Los Banos camp was conducted by joint forces of American and Filipino forces including ROTC Hunters, Hukbalahap 48th Chinese Squadron, President Quezon's own guerrilla unit, Marking's group and other irregular troops on February 23, 1945.

The Department of Human Kinetics of the College of Arts and Sciences which administers the physical education courses of the university was based there.