I don’t know about other men but speaking for myself, although my eyes often stray towards deep cleavage lines and exposed breasts, the reason isn’t sexual. I will also be distracted, temporarily, by a fine masculine body or exposed penises even though I am not homosexual.
I suppose it is cultural. Nudity is generally taboo in “civilised” society and it is uncommon, if not illegal, for people to leave their houses without their clothes on. This makes any exposure of human flesh, beautiful or otherwise, distracting.
Besides, I prefer to squeeze real breasts, those balled delicacies of tender creamed flesh. There is nothing sexy about silicon and rubber.