In 1953, U.S. Representative Hubert B. Scudder opened a Congressional hearing to determine whether the murals should be removed for themes "inconsistent with American ideals and principles"; the often contentious proceedings concluded with their retention.

The building is now part of Rincon Center, remodeled as shops and residences after the Post Office left in 1979, and was added to the National Register of Historic Places that year.

[9] Refregier was selected by a four-person jury: the Annex architect, Gilbert Underwood, and three peer artists,[6] consisting of Victor Arnautoff, Arnold Blanch, and Philip Guston.

[8]: 22  Shortly after he was awarded the contract, in 1942 Refregier told the San Francisco Chronicle he wanted paint the past as it had affected the present conditions of depression, strikes, and war.

[8]: 63  The project sponsors requested 92 changes during the design and painting of the murals, ranging from slimming a Spanish priest (#6, "Preaching and Farming at Mission Dolores")[13] to raising picket signs so their pro-union messages could not be read (#14, "Torchlight Procession").

"[17] The History of San Francisco murals created a heated debate because they depicted controversial events from California's past, painted in a public building using taxpayer funds.

[18] In response, the Public Building Administration ordered that panel covered, a decision that drew protest from the CIO longshoremen[20] and artists' organizations.

[13] In his private notes, Refregier despaired: "The stories in the Hearst press brought out gangs of hoodlums who were constantly under my scaffolding and I no longer worked after the sun set.

"[7]: 75 That fall, Waldo F. Postel and his colleagues in the Native Sons of the Golden West announced they would begin campaigning for the removal of the murals: "Just what sort of paintings are these?

Under questioning from Subcommittee Chairman James C. Auchincloss, Jackson admitted he had "seen photographic duplications" but had "not personally seen the murals" before calling them "not truly representative of the history of California", adding "if they were in the Capitol of the United States I would join in protesting them."

[8]: 10–17  Scudder then passed around black-and-white photographs of the murals and provided specific criticism for each one, singling out the depiction of indigenous people ("strong features, muscles, and physique ... [not] a true picture of the aborigines of the West"), Spanish explorers and priests ("big-bellied", "warlike", and "objectionable to people who appreciate ... those who developed California and brought civilization to the west coast"), pioneer settlers ("cadaverous, soulless Americans" and a "moronic assemblage of people"), and gold miners ("depicting the thing which the Communists claim, we are only seeking the golden riches in our mode of life"), among others.

It is injurious because it would mean the destruction of what, to judge from my recollection of your sketches and from reproductions of the finished murals, is a remarkable work of art, and an outstanding example of the growing tendency in your country to try to exert political control over freedom of thought and expression, and to impair the liberty of the creative artist ...

The lamentable state of biology and philosophy in the U.S.S.R. shows what happens when creative thought and expression is subjected to control on political or ideological grounds.

Later in the defense, a statement from Warren Howell was read into the record, providing "recognizable sources which are authoritative and authentic" for many of the scenes depicted in the murals.

... Judging either the painter or the style of art used would be putting the Congress in the same position as the totalitarian governments who refuse to allow music to be played if the composer's politics do not suit them.

[29] Scudder won re-election and continued to insist the murals were "very obnoxious to people in the area", claiming that an analysis showed parts of them were "definitely Communist propaganda" in 1957.

[31] After the Post Office moved the mail distribution facilities to India Basin, vacating the space in 1979, preservationists including Emmy Lou Packard rallied again to save the murals;[7]: 89  the building was added to the National Register of Historic Places that year.

[5] The mural panels depicted various historical events from California's past, and was meant to span all of human history, from an early Native American creating art (#1, planned title: "In the Beginning, Waters Covered All Earth Except Mount Diabalo [sic]") to the Golden Gate International Exposition (#26, planned title: "Chinatown—The Fair, 1946").

Refregier "believed that art must address itself to contemporary issues and that a mural painting in particular must not be 'banal, decorative embellishment', but a 'meaningful, significant, powerful plastic statement based on the history and lives of the people'".

His painting style appears to be very rudimentary and simple, but complex because of the way he uses color to evoke emotion and powerful images to tell a story.