I think doffy is admiral level...He doesn't seem scared of them...kaido is the only dude he seems kind of scared of...and even then it more so seems like I'd rather not have to fight him and just make money off of him, but if I do
Kuzan basically made Doffy take his ball and go home, as for Fujitora the only reason he isn't scared is because of the deal they made to work together we all saw how shook Doffy was acting when Fuji dropped a meteor which was just a warm up.
When it comes to the big boys Doffy doesn't want that work.