Maybe I missed it, but does the show ever explain how some wildlings got south of the wall?? Book brehs, if its a spoiler, then nevermind
Think of Wildlings as Illegal Immigrants. The Night's Watch are few and are spread thin. So they can't catch em all.
Night Watch = Border Patrol
Winterfell = ICE