I have a theory.... there's only ONE Kang.
He let silvey kill him to free him. He probably was trapped and that castle was a prison. he waited until their was a tear in the timeline (both loki's helped facilitate that) to throw off everything with his "help" because he knew how most of everything would play out. But once he reached the point of no return, he needed someone to kill him to free him.
OR this kang was imprisoned by the other Kangs... He killed most of them, and the remaining few lelft him in that prison.