Littlefinger's downfall was his love of Stark women, gave Sansa leverage when she lied for him about her aunt's death and the Vale dudes were dead set on killing him. He also sold her to the Boltons' which turned her away from him when she came to see her and admitted that his life was in her hands. He is smart but made miscalculations along the way which left him pledged to the Starks and they had a hold on him. I think the Vale people never liked him anyway. A lot of the players have gone and it's now down to Dany vs Cersei with not many allies left, LF did not have room to maneuver in that new reality.