I think reduced economic mobility plays a part as well.it's always folks who squandered the opportunities given them by their luck to have born in America
who are most upset at the idea of other people risking their lives to get access to a fraction of that opportunity
The more people begin to see the economy as a fixed pie that needs to be divided up the more the idea of adding more mouths to feed becomes undesirable.