It's both aspects (Black and man) but more so being Black.
Unfortunately we're already seen as "threatening" and the media depictions don't help so a lot of non-Black people would have already made up their mind about us.
I say this because even suited and booted up brehs tend to get the same treatment, I certainly have.
I sometimes wonder how the women act when they see a cac or an Asian man walking up towards them. Do they feel at ease? Do they let their guard down a bit?
I also wonder whether colorism plays a hand in it (which links back to the above paragraph) since the darker you are, the more "threatening" you're perceived as.
All you can do is not internalise it, we just have to play the cards we are dealt even though it's not deserved.
When you see a female coming the other way at night, cross the street and give her room so that both parties can feel at ease
.