The freedom of speech and the freedom of contracting are analogous in that they both bolster innovation: of ways of life, institutions, goods, and services. Indeed, the freedom of speech may be moot without the freedom of contracting. The purpose of speech is to coordinate individuals on acting differently, on contracting differently.
Brennan and Jaworski make a point that is rather obvious but apparently needs to be reiterated to the self-proclaimed moral philosopher: one cannot build a theory of morality by merely consulting one's prejudice (which is culture-specific) and instinct (which is animalistic). Nor can one make factual statements without consulting facts. Moral philosophy is a branch of social science, whose scientific method it must respect lest it never rise above demagoguery.
Like most men, moral philosophers are ill-equipped to ponder trade-offs. An act is either good or evil, moral or immoral. Rawls advocated the minmax criterion, presumably, because he could not conceive of a way to trade-off one individual's wellbeing against another's. Some moral philosophers today similarly castigate some acts as being so disgusting (to the refined observer) as to warrant an unqualified ban, even if its imposition costs lives. Such a dichotomous approach to morality might have been a decent rule of thumb for the troglodyte, but the modern man is intelligent enough to afford a more nuanced approach.