THIS LOOKS ridiculous on me,” thinks Brian Madigan when he attempts to wear bright colors. It’s not that he dislikes them; he envies friends who can toss on a maroon shirt or a lemon-hued polo without hesitation or self-consciousness. “I wish that I could have an eye for the types of color they’re wearing,” says Mr. Madigan, 36, a photo editor in San Jose, Calif. But he feels far more comfortable in his familiar gray T-shirts, blue jeans and tan khakis. Better safe than startling.

Many American men occupy Mr. Madigan’s chromatic…