The Monterey Colonial style house was built 1838-41 by General José Antonio Castro, a former Governor of Alta California.

It was later sold to the Breen family, who lived there until 1933, when the house became a museum as part of San Juan Bautista State Historic Park.

José Antonio Castro used the house as an administrative base for his military operations (soldiers' barracks were next door) and let his secretary use it as a residence.

The home is clad in stucco and includes a second-story full-length covered porches on both long sides, characteristic of Monterey Colonial architecture.

Nowadays, the José Castro House property includes a half-acre orchard and garden, open to the public as part of the state historic park.