Too much or too little wrestling does matter. Sure there are probably no "casual" fans based on your definition for what a casual fan is, but plenty of people watch wrestling for different reasons and not having a healthy mix of everything can be a turnoff.
Some people watch for characters/gimmicks
Some people watch for promos
Some people watch for storylines
Some people watch strictly for wrestling
Some people eyes are glued based on some of the comedy segments
Some people eyes are glued based on some of the serious segments
Some people watch for everything
etc
I am one of those people that mainly watch for the characters, promos, and storylines. The wrestling added to it is just the perfect mixture and is why I am a fan. I stopped watching WWE/AEW consistently for the past 2 months because storylines were shytty to me and not really good. I can't watch a wrestling match without the character being part of a good storyline that is going to further progress their development.
Especially nowadays where I feel like I have seen all the characters moves and their not doing anything special in their matches to surprise me. That is when the story element to a match plays a huge factor. That is why some of these wrestlers should slow tf down and stop giving us everything at once. It leaves no room for surprises.