Franklin Roosevelt actually said...
No passport shall be granted or issued to any other persons than those owing allegiance, whether citizens or not, to the United States.
Context
Executive Order 7856—Rules Governing the Granting and Issuing of Passports in the United States
03/30/1938