[3] On May 24, Confederate Maj. Gen. Fitzhugh Lee's cavalry division (about 2,500 men) attacked the Union supply depot at Wilson's Wharf, on the James River in eastern Charles City, Virginia.

During the winter of 1863–64, Wild led these soldiers in an expedition on the coast of North Carolina, terrifying a local white population accustomed to African slavery since the early 18th century.

[4] Wild's brigade landed in Virginia in May 1864 and began building the fort at Wilson's Wharf, one of a series of protective outposts guarding supply lines for Union Maj. Gen. Benjamin Butler's Bermuda Hundred Campaign.

The wharf was at a strategic bend in the James River, overlooked by high bluffs, 2 miles (3.2 km) from Sherwood Forest, the home of former U.S. President John Tyler.

[5] Succumbing to the political pressure, Davis's military adviser, Gen. Braxton Bragg, ordered Fitzhugh Lee's cavalry division to "break up this nest and stop their uncivilized proceedings."

[7] Around noon on May 24, Lee's men charged and drove in the Union pickets who were posted near the Charles City Road, about a 1 mile (1.6 km) north of the fort.

Wild and his men interpreted this to mean they would be killed or enslaved, particularly due to the massacre of black troops after their surrender at the Battle of Fort Pillow six weeks earlier.

[10] As Lee looked for a weak point in the fort's defenses, Union reinforcements arrived at about 4 p.m. on the steamer George Washington, carrying four companies of the 10th USCI.

Southerners, unwilling to acknowledge their defeat against a predominantly African-American force, claimed that six gunboats and substantial numbers of white Union soldiers were involved in the action.