They thought they'd won the moment Frodo was stabbed, since his wound would have eventually turned him into a wraith. As for Aragorn, the ringwraiths' "chief weapon" is their ability to cause debilitating fear in others; the fact that aragorn was unafraid, ready to fight, and swinging around fire was a level of resistance they were not expecting. Given that it was only a matter of time before Frodo succumbed to his wound, they withdrew thinking they just had to wait it out a bit.
As for the Witch King specifically, Sauron gave him some of his own power in the run up to the siege of minas tirith. So while he was definitely stronger than he was in Fellowship, he was not strong enough to casually break Gandalf's staff. That only happened in the movie and I fukking haaaaate it. In the books, they just have a stand-off until Rohan arrives and interrupts them before they can fight. In truth, Gandalf the White would've fukking washed him and anyone/thing else in middle earth besides Sauron.