If morals are man-made, then they are merely preferences, not morals at all. While this assertion excites some, the truth is that this distinction carries with it consequences.
We're not just talking about sexual morality. We're talking about everything from the nasty extremes (e.g. murder and rape) through the mild "no big deals" (e.g. white lies and line cutting).
You cannot reasonably expect my preferences to match yours, and if you try to impose them upon me, then you are a tyrant who must resort to violence in order to achieve your ends. Without morality in the traditionally accepted and defined sense, there is no such thing as right or wrong, good or evil.
If you won't have morality as objective and trans-ego, then you cannot have morality at all. As T.S. Eliot observed, "If you will not have God you should pay your respects to Hitler or Stalin."