Hillyer's father was the postmaster of Henderson, Kentucky; his mother was a niece of Benjamin Silliman, a noted scientist and an educator at Yale University.

Hillyer lost both his parents when he was a youth, and along with his older sister Elizabeth, went to live with their aunt Mary Lapsley in New Albany, Indiana.

Hillyer married Anna Rankin of Newark—together they had six children[4] which include twin sons whom he named after Generals Grant and Rawlins.

He often wrote to his wife concerning personal matters and other activities, with other correspondence to his friends and acquaintances and to a number of military personnel including Ulysses S. Grant.

[6] He was employed as a real estate agent renting office space from Grant and Harry Boggs.

Soon after Ulysses Grant was commissioned as brigadier-general and in August 1861[9] he offered Hillyer from his former regiment a place on his staff, with the rank of captain.

[10] On October 4, while stationed in Cairo, Illinois, Rawlins by Special Order, appointed Hillyer as the Mustering Officer for the district in compliance to Lincoln's calls for volunteers.

"[15][16][17] The entire dispatch was written and carried by Hillyer who delivered it in person to Confederate General Buckner, commander of the fort.

[19] In April 1862 Governor Hamilton Gamble of Missouri appointed Hillyer as aide-de-camp with a promotion to the rank of Colonel of Volunteers on May 3, 1862.

[9][21] Hillyer left the battlefield to bring Grant's several regiments from Savannah back to Pittsburg Landing, where they were desperately needed.

In March, 1865 he was brevetted brigadier general and in June served as the chairman of the Grand Reunion of the Army of the Tennessee.

[9][c] In 1868 Hillyer was appointed a U.S. revenue agent by President Andrew Johnson until the position was abolished by Congress, after which he served as a lawyer for the Commissioners of Immigration.

[26] While he was bedridden at the Owen House during the last three weeks of his life President Grant was a daily visitor at his bedside.