i wouldnt be surprised. most show watchers that i know are confused enough as it is about all the characters and their names. introducing the manderlys, the rest of the martells, the greyjoys, everybody brienne meets, the new high septon and that whole storyline, brown ben plumm, jon connington and aegon and their group, the golden company, the mountain clans that stannis recruits, all the freys that come up north with the boltons, the people in braavos when arya moves out there, the lannisters that jamie runs into in the riverlands, alys karstark and her uncle, etc etc etc would confuse the shyt out of them. it would be too much. they gonna have to combine and compress a shyt load of storylines for the next 3 seasons.
EDIT:
i didnt even mention lady maege and her daughters, the glovers, and the rest of the vale