The system began to break down when the Congress of the Confederate States of America classified African-American prisoners of war as fugitive slaves on May 1, 1863, who ought to be returned to their owners instead of being exchanged.
In the same act, captured white officers of armed "negroes or mulattoes" were withheld as inciters of a "servile insurrection", being threatened with legal prosecution up to and including the death penalty.
The Lincoln administration wanted to avoid any action that might appear as an official recognition of the Confederate government in Richmond, including the formal transfer of military captives.
In the North, public opinion on prisoner exchanges began to soften after the First Battle of Bull Run, when the rebels captured about one thousand Union soldiers.
[5] Prior to the cartel's creation, Union and Confederate forces exchanged prisoners sporadically, usually as an act of humanity between opposing field commanders.
[5] On December 11, 1861, the U.S. Congress passed a joint resolution calling on President Lincoln to "inaugurate systematic measures for the exchange of prisoners in the present rebellion.
[12] In June 1862, General Cobb met with Union Col. Thomas M. Key, an aide to Maj. Gen. George McClellan, in another attempt to reach an agreement on prisoner exchanges.
In addition, the agreement permitted each side to exchange non-combatants, such as citizens accused of disloyalty, and civilian employees of the military, such as teamsters and sutlers.
The terms of the cartel prohibited paroled prisoners from returning to the military in any capacity including "the performance of field, garrison, police, or guard, or constabulary duty.
Gen. Lorenzo Thomas, conducted their first official prisoner exchange under the agreement's terms with a transfer of 3021 Union personnel for 3000 Confederates at Aiken's Landing.
Four weeks ago the Federal Agent informed me in writing that it was not the intention of his Government to make any more arrests of non combatants in our territory, and yet more have been made since that declaration than during any previous equal space of time.
In November 1863, Union General Benjamin Butler requested permission from Secretary of War Edwin Stanton to negotiate for the resumption of the prisoner exchanges.
Despite his original mandate, Butler tried to resolve the outstanding cartel issues with the rebel authorities while facing General Hitchcock's growing opposition over the scope and conduct of his activities.
[23] Asked to review the situation in April 1864, Union General Ulysses S. Grant ordered the halt of all exchanges until the Confederates recognized "the validity of the paroles of the prisoners captured at Vicksburg and Port Hudson," and stopped discrimination against "colored soldiers.
[26] In January 1865 with the end of the war in sight, General Grant permitted the resumption of exchanges when the Confederate authorities agreed to include all prisoners.