Agee was nominated on March 13, 2008 by President George W. Bush to fill a vacancy on the Fourth Circuit created by Judge J. Michael Luttig, who resigned on May 10, 2006.

President Bush asked the Senate to consider his nomination swiftly because of the court’s heavy caseloads, and because five of the fifteen seats were vacant.

In 2016, Agee found that sectarian prayers offered by Rowan County, North Carolina commissioners at their meetings did not violate the Establishment Clause of the United States Constitution, over the dissent of Judge Wilkinson.

That judgment was then rejected by the full circuit en banc by a vote of 10-5, with Wilkinson now writing for the majority while Agee and Paul V. Niemeyer authored dissents.

[4][5] In June 2018, the Supreme Court of the United States denied review, over the written dissent of Justice Clarence Thomas joined by Neil Gorsuch.