"[2] Agee and his collaborators repeatedly and publicly identified individuals and organizations in foreign countries as undercover CIA agents, employees, or sources.
[5] It held that the Secretary was required to show that Congress had authorized the regulation either by an express delegation or by implied approval of a "substantial and consistent" administrative practice, citing Zemel v. Rusk (1965).
The Court of Appeals took note of the Secretary's reliance on "a series of statutes, regulations, proclamations, orders and advisory opinions dating back to 1856," but declined to consider those authorities, reasoning that "the criterion for establishing congressional assent by inaction is the actual imposition of sanctions and not the mere assertion of power."
The court also regarded it as material that most of the Secretary's authorities dealt with powers of the Executive Branch "during time of war or national emergency" or with respect to persons "engaged in criminal conduct."
Matters intimately related to foreign policy and national security were rarely proper subjects for judicial intervention particularly in light of the "broad rule-making authority granted in the 1926 Act," citing Zemel.
The Court rejected Agee's argument that the only way the Executive could establish implicit congressional approval is by proof of longstanding and consistent enforcement of the claimed power.
The Court found that the right to hold a passport is subordinate to national security and foreign policy considerations, and is subject to reasonable governmental regulation.
Denial of Agee's passport was not protected under the First Amendment because unlike Kent v. Dulles and Aptheker v. Secretary of State involving denials of passports solely on the basis of political beliefs entitled to First Amendment protection, Agee's actions amounted to more than speech and that the national security interests here, as in Zemel v. Rusk, were sufficiently important to justify revocation.
Blackmun concurred, stating simply that he believed this case had cut back somewhat on the standards articulated in Zemel v. Rusk and Kent v. Dulles sub silencio and that it would have been better if the Court had forthrightly construed the law such that evidence of a longstanding Executive policy or construction in this area be probative of the issue of congressional authorization.
Those cases held that an administrative practice must be demonstrated; Kent unequivocally stated that mere construction by the Executive - no matter how longstanding and consistent - was not sufficient.
The decision applied not only to Philip Agee, whose activities could be perceived as harming the national security, but also to other citizens who may merely disagree with Government foreign policy and express their views.
The point that Kent and Zemel made, and the majority opinion should make, is that the Executive's authority to revoke passports touched an area fraught with important constitutional rights, and that the Court should therefore "construe narrowly all delegated powers that curtail or dilute them".