James Madison actually said...
I recommend also that further restraints be imposed on the issue of patents to wrongful claimants, and further guards provided against fraudulent exactions of fees by persons possessed of patents.
Context
Special Message
04/10/1816