James Madison actually said...
As long as the reason of man continues fallible, and he is at liberty to exercise it, different opinions will be formed.
Context
On the inevitability of differing opinions in a free society.
11/21/1787