Castello Brown is a historic house museum located high above the harbour of Portofino, northern Italy.

Fortifications from the fourth century AD have been found underneath the modern castle;[1]: 63  a castrum and a turris are recorded.

[1]: 61 As Portofino was an important harbour, the Castello featured in many naval battles between the 13th and 19th centuries, withstanding attacks by the Genoese (Ghibelline) admiral Aitone D'Oria in 1330, by a Venetian fleet in 1431, and by the British Navy in 1814.

According to the Record Office of Genoa, cannon batteries were constructed on the site in the early 16th century, and military engineer Giovanni Maria Olgiati [it] drew up plans for a full fortress circa 1554.

Napoleon garrisoned the Castello and increased its armament to protect coastal traffic against the British fleet, which unsuccessfully attacked it again in March 1814.

[1] He engaged the architect Alfredo D'Andrade [it], and with advice from his artist friend and fellow-consul James Harris the fort was transformed to a comfortable villa without substantial alteration in its general form.