To me, it's always been aesthetic. Milk is little more than liquid snot, and to drink it by the glass has struck me as revolting for over thirty years. It's just a bonus to be "un-American" by rejecting milk in liquid form, although I've returned to eating cereal with soy milk as moistening agent.
I adore cheese, cream-based sauces, dairy-laden desserts and Milk Stout; obviously, I can tolerate lactose, but drink it from a glass?
That's just wrong. Yuck.
Got Milk? You Don’t Need It, by Mark Bittman (Opinionator; New York Times)
... Today the Department of Agriculture’s recommendation for dairy is a mere three cups daily — still 1½ pounds by weight — for every man, woman and child over age 9. This in a country where as many as 50 million people are lactose intolerant, including 90 percent of all Asian-Americans and 75 percent of all African-Americans, Mexican-Americans and Jews. The myplate.gov site helpfully suggests that those people drink lactose-free beverages. (To its credit, it now counts soy milk as “dairy.”)
There’s no mention of water, which is truly nature’s perfect beverage; the site simply encourages us to switch to low-fat milk. But, says Neal Barnard, president of the Physicians Committee for Responsible Medicine, “Sugar — in the form of lactose — contributes about 55 percent of skim milk’s calories, giving it ounce for ounce the same calorie load as soda” ...