There's something about the way the show is handled.... I can't put a finger on it. The art, music, pacing.... it's hard to describe. The show is flat out perfect for what it's about? Like you know it's not gonna appeal to everybody, but they wanted to make the show perfect for what it is. and they succeeded.
I said earlier that it's like Akagi or Saki, where it takes something you have absolutely no knowledge of (actual mahjongg for the mentioned anime, karuta here) , but the passion the characters put into the game makes you want to learn it.
The sound of a player smacking the mat to draw a card is an example of that. It's a subtle detail that you don't pay much attention to in the moment, but when you hear those loud thumps it adds to your immersion into the matches and lets you know how much the players care about Karuta without ever needing to hear them say so.
Then as a viewer you may think, what do I have in my life I'm
this passionate about. Something that a lot of people would look at me crazy for being obsessed with but I don't give a damn when I'm absorbed in the moment of the activity. If you have something like that then you relate instantly. If not then you maybe admire and are inspired by them to find that thing. Either way these characters are designed to be endearing and I think that's the foundation of Chihayafuru's appeal. Same with lots of other sports anime that can convince you to care.
Imo a LOT of people who will never watch Chihayafuru would love this show. But it's a matter of exposure and preconceived bias against the genre. The romance tag doesn't do it any favors when its much more than the usual crap produced in the genre.
Haven't seen Akagi (have seen Kaiji though), but it does remind me of how the characterization in Ippo works to make you appreciate boxing regardless of how you feel about it going in. Doesn't really matter what the activity is as much as how the characters can sell that activity as aided by the animation and soundtrack of the show. I'm barely through it at this point, but I'm already thinking it has a place in my top 20 all time just b/c of how sincere it is. Maybe something stupid happens later but its great now.