James Monroe actually said...
As Colonel Gadsden held the office of Inspector-General, and as such was acknowledged by all to belong to the staff of the Army, it is not perceived on what ground his appointment can be objected to.
Context
Special Message
04/05/1822