The palace is located on the corner of Kralja Milana and Dragoslava Jovanovića streets, opposite Novi Dvor (seat of the President of Serbia).

Simić drained the marsh, filled and leveled the terrain and on the northern side of the modern Kralja Milana street constructed a house in 1842.

Development of the first Serbian royal compound began in 1843, when the ruling prince Alexander Karađorđević purchased the konak with the surrounding garden.

Upon ascending to the throne in 1860, Prince Michael decided to use the Stari Konak, while part of the state administration (ministries of foreign and internal affairs) was located in Palace with the Towers.

[2][3] Instigated by the King Milan, Stari Dvor was built between 1882 and 1884, according to the design of Aleksandar Bugarski, in the style of academism of the 19th century, with intention to surpass all previous residences of the Serbian monarchs.

[2] English author Herbert Vivien, who visited the palace by the end of the 19th century, described in detail its interior: "At the left side, there is a fine ball room, with walls of lemon-yellow colour, with huge white lusters of Venetian glass, glistening nicely during the state festivities, lit by electric light.

[2] In 1919 and 1920, meetings of the Provisional National Assembly of the newly-formed Kingdom of Serbs, Croats, and Slovenes took place in the palace.

Korunović's design of the eastern façade was mostly plain, but the western, facing the court, was richly decorated with ornaments.

Soon, the reconstruction of the building began but wasn't finished until October 1944 when Belgrade was liberated as the dome laid in ruins.

There were suggestions that monument to King Alexander I Obrenović, erected in 2004 along the boulevard bearing his name, should be relocated to the location of the 1904 demolished Stari Konak, where he was assassinated.

There used to be a greenhouse and richly ornamented oaken stairs added later, and leading onto the first floor (they were designed by a famous architect Jovan Ilkić).

The caryatid are repeated on the facade facing the Kralja Milana street, and the line of Doric columns in beneath them.