For statistical purposes, the United States Census Bureau has defined Los Olivos as a census-designated place (CDP).

Sometime around 1880, on a bluff overlooking Alamo Pintado Creek, just north of the town of Ballard, a two-story house was built, with a wide, covered front porch and neatly symmetrical arched windows in the center gable, situated on prime farmland.

It became the property of twenty-two-year-old Alden March Boyd, of Albany, New York, when he paid $8,000 for "157 acres, more or less, together with the dwelling house," in 1885.

He planted five thousand olive trees, and called it Rancho De Los Olivos.

[7] Los Olivos was connected by the narrow gauge railroad to points north as far as San Luis Obispo until the train made its last run in 1934.

The southern terminus of the railroad was in front of Mattei's Tavern, where a stagecoach line continued over San Marcos Pass into Santa Barbara.

Starting in Los Olivos and stretching north is the Foxen Canyon Wine Trail.

Several locations in Los Olivos were featured in the 2004 movie Sideways highlighting Santa Barbara county's viticulture.

Portions of the 1983 video for the song "Say Say Say", with Michael Jackson and Paul McCartney, were filmed at Sycamore Ranch, five miles (8 km) from the town.

[14][15] US President Ronald Reagan lived about 9 miles (14 km) south at Rancho del Cielo.

[16] Musician Michael Jackson's property Neverland Ranch is located about 5 miles (8.0 km) north.