At their southwestern margin, the Glendale Narrows section of the Los Angeles River separates them from the eastern tip of the Santa Monica Mountains.

The San Rafael Hills contain all or parts of the communities of City Terrace, La Cañada Flintridge, Pasadena, South Pasadena, El Sereno, Monterey Hills, Montecito Heights, Cypress Park, Mount Washington, Glassell Park and foothills surrounding Eagle Rock, east of the Glendale Freeway including Rancho San Rafael and Chevy Chase Canyon in Glendale.

They were the homeland, with settlements, of the Tongva Native American people for over 8,000 years before the Spanish invasion and colonization of the late 18th century.

The hillside campus of Art Center College of Design, a Pasadena historic resource, is located in the San Rafael Hills.

Flintridge Sacred Heart Academy, a private all-girls Catholic high school, sits atop the crest of the San Rafael Hills.