In the north, Hirschhorn borders the villages of Heddesbach (Rhein-Neckar-Kreis, Baden-Württemberg) and Brombach (part of Eberbach), and on the parish of Rothenberg (Odenwaldkreis).

This settlement, which in 1023 under the name of Erasam belonged to the property of a monastery affiliated to Lorsch, St Michael's on Heiligenberg near Heidelberg, was one of the oldest in the Neckar valley.

Whereas almost the whole of the surrounding area came into the possession of the diocese of Worms in the 11th century, Ersheim together with the nearby village of Ramsau downriver on the right bank remained an exclave of Lorsch.

From here several villages were founded in forest clearings from the 12th century onward, among them Weidenau, Unter-Hainbrunn, Igelsbach and Krautlach, but they were largely abandoned again afterwards.

Engelhard I (1329–61) increased his influence and his dominions considerably through Imperial fiefdoms and land mortgaged to him in return for loans.

When Elector Palatine Ruprecht III was elected King in 1400, Hans V of Hirschhorn was employed in Imperial service as an adviser, diplomat, and financier.

King Henry V of England held Hans in such high esteem that he granted him a lifelong annual payment of 100 marks.

The inhabitants of the neighbouring villages sought protection within the precincts of the fortified town, so Ersheim, Ramsau, Krautlach and Weidenau were soon abandoned.

After the Lords of Hirschhorn had died out with the demise of Frederic III in September 1632 - he had fled to Heilbronn in order to escape the turmoil of the war - the castle and the town passed to the archdiocese of Mainz.

After the end of Swedish occupation in 1636, Hirschhorn was mortgaged to an official at the court of the Elector of Cologne, Rudolf Raitz von Frentz.

Following the Peace of Westphalia (1648), new inhabitants from Palatinate, the Electorates of Mainz and Trier, Lorraine, Tyrol and Switzerland settled in the town.

Between 1676 and 1699, Hirschhorn was mortgaged to Westphalian baron Johann Wilhelm von der Reck, but in 1700 direct rule by the Electorate of Mainz was established.

"[18] The city of Hirschhorn in the Bergstraße district was approved by the Hessian Minister of the Interior on August 5, 1964, to change the previous coat of arms.

In 1844 Joseph Mallord William Turner, the famous English Romantic artist, painted some watercolours of several places in the Neckar valley.

Set against the background of the Thirty Years' War, Das Deutsche Herz gives a vivid impression of the age of chivalry in its decline.

He travelled from Heilbronn to Hirschhorn by boat, stayed overnight at the hotel "Zum Naturalisten" on August 9, 1878, and continued his journey to Heidelberg by coach and train.

In his book A Tramp Abroad, the boat becomes a raft, and the travellers end up in Hirschhorn after a terrible storm on the Neckar from which they just manage to escape.

"I dozed off to sleep while contemplating a great white stuffed owl which was looking intently down on me from a high perch with the air of a person who thought he had met me before but could not make out for certain."

Then the clustered brown towers perched on the green hilltop, and the old battlemented stone wall stretching up and over the grassy ridge and disappearing in the leafy sea beyond, make a picture whose grace and beauty entirely satisfy the eye."