To the north is Union Bay; to the west are Montlake and Madison Valley; to the south is the Washington Park neighborhood; and to the east is the Broadmoor Golf Club.

State Route 520 cuts through Foster Island and the Union Bay wetlands at the park's northern end, interchanging with Lake Washington Boulevard just outside the arboretum entrance.

A footpath winds underneath the freeway overpasses and over boardwalks, along the Lake Washington ship canal, and into the gardens of the Arboretum.

It also receives runoff from Rhododendron Glen and the Woodland Garden, as well as sub-surface drainage from the neighboring course of the Broadmoor Golf Club.

They called it Sxwacugwit, meaning "portage" or "narrow passage" due to its strategic importance as a transportation link between the coast and the inland river systems which fed into Lake Washington.

[3] In the 19th century, the land was purchased from the federal government by Fred Drew, a timber surveyor working for the Puget Mill Company.

The eastern 200 acres (0.8 km2) were developed as the Broadmoor Golf Club by a group of businessmen that included E. G. Ames, general manager of Puget Mill.

Architect Victor Steinbrueck, writing in 1962, objected to the "naked brutality of unimaginative structures such as this proposed crossing of Portage Bay, which eliminates fifty houseboats while casting its shadow and noise across this tranquil boat haven.

[12] The potential impact of plans to reconstruct and expand State Route 520 and replace the Evergreen Point Floating Bridge in the early 2010s raised concerns among Arboretum staff and park users.

[13] As the members of the Arboretum community noted in their collective letter to the Washington State Department of Transportation, "Native plants, wetlands, and wildlife ... would be affected not only by the taking of land but by the looming shadows created by roadways in various proposals".

[14] Among the alternative proposals was the "Arboretum Bypass Plan," to build the new elevated highway over Union Bay on a more northerly route than the then-existing one.

Park improvements included a new multiuse trail from East Madison Street to Montlake and the University District and a pedestrian undercrossing on Foster Island.