James Madison actually said...
If an unworthy man be continued in office by an unworthy President, the House of Representatives can at any time impeach him and the Senate can remove him whether the President chooses or not.
01/18/2024