I think both of you are correct.
Gus had Max, who was killed. Then he had Gale and Walt.
After Jesse killed Gale it's not as if Gus could pull another chemist out of thin air. If Jesse was on board I have zero doubts Gus would've killed Walt.
However, had Walt not been wilding immediately in season 4, I don't think Gus would've pressed the issue at all. After Gale died, if Walt was a model employee I think Gus would've let him work until his cancer killed him.
As it stands, he tried to talk Mike into turning on Gus and showed up to his house with a pistol. Let alone everything else he did over the course of the season.
After that there was no way in hell Gus was gonna let him slide.
Fred.
If I'm not mistaken, Gus even stated about waiting for Walt's cancer to return and him eventually dying. But he also wanted to kill Walt for all the shenanigans Walt was doing. With Jesse, he realized that at the very least Jesse is loyal to a fault.