Its one-story building, designed in the Greek Revival style by Scottish architect George Ashdown Audsley, stood at the corner of Mason and Jefferson streets, in downtown Milwaukee.

[1] The bulk of the gallery's works consisted of Layton's personal collection of European and American paintings and sculpture, assembled during the five years preceding the institution's opening, as well as subsequent purchases through an endowment.

[2] Later acquisitions by the gallery included purchases and gifts of works by Winslow Homer, Jules Bastien-Lepage (Le Père Jacques, 1881), Frederic Leighton, Albert Bierstadt, Lawrence Alma-Tadema, Thomas Moran, Abbott Handerson Thayer, Mihály Munkácsy, and Sofonisba Anguissola.

[5] Beginning in the 1870s, the idea of establishing a public art gallery was increasingly supported by Milwaukee's city leaders, along with the need for a permanent exhibition venue.

[10] Soon, the Milwaukee Sentinel reported that Layton "was now going abroad and intends studying the architecture and management of art institutes while there and hoped to pick up some information that would be of value in the construction of a model building.

[13] The gallery's entrance was designed as a grand portico of simplified fluted Corinthian columns, with a frieze and facade ornaments made of terracotta, while the three remaining exterior walls were built using local Cream City brick.

He attended the New York estate sales of Alexander Turney Stewart and Mary J. Morgan, at which he purchased landscape scenes by painters John Constable and Régis François Gignoux, then crossed the Atlantic to pursue his acquisitions in Europe.

[21] The total cost for construction amounted to $115,000 (roughly $3.8 million in 2025 dollars, adjusted for inflation), to which Layton added a $100,000 endowment for the purchase of art and care of the building.

[10][22] Over the next two decades, gifts from local collectors including Frederick Pabst, Philip Danforth Armour, Edward Phelps Allis, Patrick Cudahy, William Plankinton, and John Lendrum Mitchell, brought further artworks to the gallery.

With public pressure for closer collaboration mounting, the two institutions organized a joint exhibition of contemporary Wisconsin art in 1948 to mark the state's centennial.

[28] The project of a new war memorial on the shores of Lake Michigan by architect Eliel Saarinen, succeeded by his son Eero, gave rise to renewed calls for a centralized art center in Milwaukee.

[41][42] Alumni of the school in its later years include illustrator Lois Ehlert (1957), film director Larry Clark (1963), painter Tom Uttech (1965), and land artist Roy Staab (1965).