I think everyone forgets that Jiren's extraordinary sense of awareness and strength is way beyond what anyone could've prepared for (given they only had a few hours to even get members and such).
Jiren even used his eyes to create a force field of all things against Hit's last ditch effort attack - Hit already knew this and essentially was like 'fukk it' and sent the Saiyans away knowing their attacks would be meaningless like his (later) attempted one was. Some may think Hit is dumb for his decision, but let's realize he was up against a guy that was able to overpower "time" and block attacks mid-dimension.
But dude Jiren is dope to me. He's the ultimate cheat code-final boss-ivan drago-monsterly booked anime character come to life. He's no-nonense, yet validly arrogant enough to deem everyone else not worthy of his time. Thing is, he arguably can't even be considered arrogant since he's consistently backed it up so far - showing real confidence in his ability (and conquering of his weaknesses) as opposed to other DB antagonists who normally just brag it up.
There's also still an air of mystery to him as well so I'm to see what his deal is.