[3] In 1736, Philipp Karl von Eltz-Kempenich (1665–1743), the Archchancellor of the Holy Roman Empire and Prince-Archbishop of Mainz, purchased a Vukovar manor in Syrmia, in the eastern Kingdom of Slavonia, then part of the Habsburg monarchy ruled by Emperor Charles VI.

The palace was originally built between 1749 and 1751 by the Archchancellor's descendants of the German Catholic noble House of Eltz and was gradually extended over time.

The Lordship of Vukovar estates near the Military Frontier were, however, exposed to raids by Ottoman troops and local Hajduk paramilitary forces.

After the Yugoslav Partisans gained control over the country in late World War II, the manor was confiscated by the communist administration of Yugoslavia in 1944, and the family of Jakob Graf zu Eltz was forced to leave Vukovar.

Eltz Manor, however, suffered a great deal of damage during the Croatian War of Independence, when it was bombarded by the Yugoslav People's Army during the Battle of Vukovar.