That's still only one half of the puzzle though...the shyt about the b*stards and "the seed is strong" is like a Westeros version of a paternity test...and it determined that in the case of Joffrey, Tommen, and Myrcella...Robert is NOT the father...but it doesn't say who IS the pappy...I just re-watched the episode and it goes from it dawning on Ned that those ain't Robert's kids to him automatically knowing they're Jaime's with enough certainty to confront Cersi to her face about it...best I can tell none of John Arryn's clue had anything to do with Cersi/Jamie fukking, but just the illegitimacy of the kids...and neither Varys nor Littlefinger hipped him to it...and when you find out a bytch has been cheating, typically her brother aint the first person you think of as suppliers of outside dikk...she could have been having an affair with anybody
the show indicates that there were constant rumors about cersai fukking her brother. so as soon as ned finds out that the children aren't his, heremembers these rumors and figure that they are probably true.