The power dynamics are repulsive to me.
I figured it would be a power issue - I find that the women who would otherwise not be completely turned off by polyamory dislike 1 man multiple women harem ideal. Perhaps a more equal set-up or one's own set of male concubines is more desirable
Without structure or a communal sort of set-up, it is difficult to sustain. Having said that, I've met swingers that are still going strong after years. Lot of interesting people with interesting set-ups out there.
For myself, I'd prefer monogamy - it suits my natural inclinations and when I like a girl, I don't see anyone else.