Depending on the genre, watching some anime is like watching an animated graphic novel. Watching an animated story with adult themes is a different medium than live action. The generation of people who were around when arcades will still popular in the U.S. before consoles pushed them out, were also there when anime was being mass imported to the U.S. as anime, rather than just Speed Racer, Voltron, G-Force. The people who were there for the initial anime explosion supported it enough to allow the industry to grow to what it is today. The anime industry has changed a lot, and the gritty 80s/90s animes that those people grew up with, has been replaced by brighter, more commercialized anime which appeals to a younger generation. A lot of people who used to watch anime don't, unless they hear about something specific, like Attack on Titan.
Like Wink said, anime is a form of escapism, as with most forms of television and film. It can be argued that it is a bit different for Black and Latinos who are rarely depicted in anime, and if so not necessarily depicted positively. Rumiko Takahashi, who is famous for creating the Inuyasha, Uruseiya Tsura, and Ranma 1/2 manga, uses a Black face style as shown in her 1987 One-Pound Gospel manga to depict a latino boxer. Modern anime in that regard has improved, as characters who were depicted in Black face in their initial incarnations like Pyunma/008 from the 1968 Cyborg 009 franchise, are now not portrayed as such in the 2015 Cyborg 009 vs Devilman. Black characters while found in sci-fi and sports themed manga, are uncommon outside those genres .
Regardless, there is still an otaku subculture that exists nowadays that did not exist when anime was first mass imported into the U.S. This subculture is a reflection of the type of anime that is popularized. While anime fans initially just had large collection of anime, some rare paraphernalia, and a passing interest to learn the Japanese culture present in the anime they watched, this has changed drastically. Otaku are at their utmost extremist fans who have an obsession with Japanese culture, often glorifying it as superior to their own culture because they are ignorant of their own culture. The irony is while many of them can read, write, and speak Japanese, they do not know any of it's history, outside of what they have learned from manga/anime/j-dramas.
What is bizarre is that Black & Latino Otaku can be excluded and marginalized by White Otaku, despite the fact that they are into the exact same things. This is prevalent at anime conventions when Black cosplayers are criticized for not looking authentic, when even less authentic looking White ones are praised. These types of criticism are dressed up as being objective rather than racist, yet all of these Otaku disregard the racist implications of fetishizing Japanese culture. Extreme Otaku do not make up the majority of anime fans, but they are a highly visible component of them, and ultimately being an Otaku for many is just a coping mechanism for loneliness and being misunderstood.
One of the reasons why anime is able to appeal to such a large group of people is that there are several classifications within anime. There is children's anime, teen anime for boys/girls respectively, adult anime for men, adult anime for women all marketed to these demographics. There are historical, fantasy, comedy, drama, sports, crime/suspense, dark comedy, science fiction, and several other specific sub-genres that exist on in anime/manga. People who watch anime often have a specific genre that they are into, while others just watch everything. It is this ability to appeal to such a large group of people that make it so popular. This is why the Ghost in the Shell animated film, was able to get so many spin off tv series, and now have a live action film starring Scarlett Johansson.