This paper argues that a central paradox of liberal constitutionalism can be found by closely examining the freedom of association. Freedom of association includes the right not to associate; thus a regime of individual rights facilitates the creation of marginalized individuals and groups. These groups then seek social recognition, which liberal constitutionalism cannot easily grant, unless the state itself becomes complicit in discriminatory action. The existence and protection of a sphere of civil society is thus a key mechanism of both freedom and inequality. The paper uses American case law to explore this paradox, and to examine the status and likely strategy of societal pariah groups, and then examines the manner in which liberal political theory (Hannah Arendt, Nancy Rosenblum, Amy Gutmann, Iris Marion Young) address related questions.