While the state capitol building has always been located on its present site, for most of the 19th century the neighborhood was best known for the townhouses on Elk Street, then one of the most desirable addresses in the city.

Many politicians, including some of the state's governors and presidents Martin Van Buren[2] and Franklin D. Roosevelt,[3]: 70–74  lived there at different times.

Henry James would recall the neighborhood from his childhood visits to his aunt as "vaguely portentous, like beasts of the forest not wholly exorcised.

The park that gives the district its name was not actually built until the early 20th century, after larger government buildings had begun to dominate the area.

In it and the other three parks are statues commemorating George Washington and Albany natives like Civil War general Phillip Sheridan and electromagnet discoverer Joseph Henry.

Although the district has been affected by modern trends—most of the Elk Street houses are now offices for various organizations that lobby the state government—it has remained mostly intact.

The 36-acre (15 ha; 150,000 m2)[2]: 9  district is rectangular, extending a block to the north and south of Washington Avenue (New York State Route 5), with an irregularly shaped projection at its northeast corner.

At South Swan Street, it turns north, with the Alfred E. Smith State Office Building and other contributing properties of the adjacent Center Square/Hudson–Park Historic District on the west.

[2]: 13 [5]: 13 It continues north two blocks, now bordering the Washington Avenue Corridor Historic District,[6] to the Elk Street intersection.

Here it runs past Cathedral of All Saints and the back of the State Education Department building to the South Hawk Street intersection.

[2]: 13 [8]: 33–35 Just before reaching St. Mary's Church, at Steuben Street, the line turns west again, then south, between the New York State Court of Appeals Building and the parking lot behind it.

[2]: 11 In the academy building, a dozen years later, one of the school's professors, Joseph Henry, conducted experiments with electricity that proved the existence of inductance and created the first functional electromagnet.

For several years in the early 1830s he demonstrated the practical effects of this discovery to his classes by using a magnet to ring a bell at the end of a wire run around the room.

[2]: 8 Martin van Buren, a state senator and New York's Attorney General before he became U.S. President, owned 4 Elk Street and lived there for some of the time he was not serving in the latter post.

[3]: 74–78 After the Civil War, during which a temporary structure was set up in Academy Park as the Army Relief Bazaar to raise money for medical supplies, this change accelerated further with the beginning of construction of the new capitol.

[20]: 7–8 The same year, the thousand-foot (300 m) Hawk Street Viaduct was built, connecting the neighborhoods north of Sheridan Hollow, now home to many of the workers in the industrialized city, with the Lafayette Park area.

Albany society took immediate notice of his penchant for the latest clothing, his fine horses, and his English coachman, who had most recently worked for Lilly Langtry, Prince Edward's mistress.

A fire insurers' organization converted 1–2 Columbia Place, including one of the buildings that had served as sculptor Erastus Dow Palmer's studio in the middle of the century,[3]: 70–74  into its offices.

[3]: 68  In 1925, a J. Massey Rhind statue of another Albany military man, Revolutionary War General Phillip Schuyler, went up in front of City Hall.

[3]: 71–73 West Capitol Park was expanded threefold from a modest plan submitted by the sons of Frederick Law Olmsted in 1898.

A replica of Jean-Antoine Houdon's statue of George Washington, displayed in the rotunda of the Virginia State Capitol, was installed in the park in 1932 to commemorate the first president's bicentennial.

[3]: 70–74  The Pruyns had moved out of 17 Elk Street around 1910;[2]: 10  their house eventually became the state headquarters of the Independent Order of Odd Fellows.

The house at 4, where the young Henry James had visited his aunt almost a century before, likewise became home to another fraternal organization, the Moose International.

That began to change in the late 1960s as urban renewal touched Albany, and the modernist towers of Empire State Plaza rose to the south, dwarfing the older government buildings in the district.

[2]: 8  At the corner of Elk and Eagle, activists were able to save the facades of three townhouses which were otherwise to have been demolished for another modernist building, James Stewart Polshek's New York Law Center, which they serve to screen from the street.

Sculptor Merlin Szosz produced a stele in the classical Greek mode, made of pink Brazilian granite and adorned with the names of those casualties, plus a relief of a soldier holding his fallen comrade amid Southeast Asian vegetation.

The state originally instituted a curfew of 11 p.m. for Lafayette Park in order to force them to leave, and police cited several of them for violating it.

These plans failed when David Soares, the county district attorney, dropped the charges and city officials issued the group a permit to stay in Academy Park, which was under its jurisdiction and not the state's, in return for its cleanup efforts and limitations on its presence.