The Arboretum is sited on a remaining portion of the Rancho Santa Anita, one of the Mexican land grants of Southern California.

Lacy Park in the city of San Marino once was another sag pond and the precise location of the third is not known but may have been on the grounds of the Huntington Library and Botanic Garden.

Not surprisingly the presence of water and game created a permanent Native American habitation in the area and is believed to have been the location of the Tongva village of Aleupkigna.

Subsequent owners of Rancho Santa Anita were; in sequence, Henry Dalton, Joseph A. Rowe, Albert Dibblee in partnership with William Corbett and a Mr. Barker, Leonard Rose and William Wolfskill, Alfred Chapman with Harris Newmark until finally the property was sold to Elias Jackson Baldwin.

In 1875 when Elias Jackson "Lucky" Baldwin purchased Rancho Santa Anita and constructed its buildings and grounds.

Baldwin started an award-winning winery to supply tourists, sold land to settlers as well as running a private water company and brick works.

A partnership with Henry E. Huntington and the Santa Fe Railroad insured that passengers could arrive by rail from Los Angeles and other locations as well as bringing freight, such as building supplies and ship ranch produce.

Groundwater recharge ponds and a pump station has been proposed by the Los Angeles County Department of Public Works and a consortium of five foothill cities for 4 acres (1.6 ha) of the Australia section.

[4] In addition, the gardens are home to a flock of peafowl which are the descendants of the first group of birds imported by Baldwin from India.

Before the depot was built, the site was a Rancho Santa Anita railway siding on the Los Angeles and San Gabriel Valley Railroad opened in 1887.

Visitors have claimed hearing moaning coming from the house (where Lucky Baldwin's wife also died of cancer), odd cooking smells, and seeing various strange anomalies walk the gardens at night.

[16][17] A number of films have been made at the Santa Anita Depot, including the remake of Christmas in Connecticut, which starred Dyan Cannon, Arnold Schwarzenegger (also directed), and Kris Kristofferson.