His father was the jurist Nicolaus Ferdinand Haller (1805–1876), a Mayor of Hamburg.

Haller's mother Adele was a Jewish-born daughter of Amschel Oppenheimer.

Haller studied architecture in Potsdam, Berlin, Paris, and England.

[1] Haller was influenced by Charles Garnier; the architect of the Paris opera; Gottfried Semper, and the Italian High Renaissance.

[1] The city of Hamburg honored Haller with a street called Hallerstraße.