I get what you're saying. Ultimately wanting to play as a male character because you're a male is no more simple-minded or inartistic than wanting to play as a female because her VA sounds good. One person's reasons can sound intellectual in their own head but insecure or pretentious to someone else. I could care less about people's personal preference or their justification for it.
The real issue is when people use bullying-tactics to impose their own personal preference as inherently right while denigrating other's as flawed/problematic/inauthentic, as a means to restrict, monopolize, and control the art.
i.e. any mc that's not white male (or any female mc that don't got a BBL) is "pandering", then proceeding to review bomb, online harass, threaten, etc