I was arguing with my wife about this, she talking about "no way he knew, he would have stopped him". Listen, he knew what was happening, that's why he brought her that toy. He has went against Stannis on a few occasions, and he knew if he did it again, he would probably be done for. At the end of the day, he realizes that Stannis is his king. That is the one that he chose to follow, that rose him from a smuggler to a Lord, to hand. He might not agree with everything he has done, but if he keeps being defiant, 1. he will be burned in all likelihood, and 2. all the devotion, fighting, trust, and honor was for not.
He has carved out a nice life for his remaining children and wife, a life that a flea bottom born lickspittle could never dream of. He also knows that all this is a means to an end. If Stannis actually wins the throne, the realm will be the better for it because Stannis is just.