Ned also met the baby in the brothel. But that's besides the point. Ned surely wasn't intending to legitimize Gendry, a boy he met just once. He intended to lay claim to Stannis as the rightful heir. I know this cuz he states this explicitly to Renly.
Furthermore, Varys wouldn't need that letter for anything. If he's going to go down the "rightful heir to Robert Baratheon" road, Gendry's the only person left. Roberts brothers and their kids (well just the one) are all dead. The fake kids with Cersei are all dead. Gendry's his only living heir legitimized or not.
All that aside, Varys has seen nothing to make him want to support Gendry as a ruler. I don't even know why I'm in engaging in this nonsense, carry on breh.