House prices reflect desirability of an area. There are plenty of places that are affordable in America they are just not gonna be downtown Miami and 2000sqft.
Take this Brooklyn listing for example
2071 Haring St, Brooklyn, NY 11229 is for sale. View 13 photos of this 2 bed, 2 bath, 1830 sqft. single family home with a list price of $1400000.
www.realtor.com
You couldn't pay people to live there 20yrs ago but now its worth over a million.
This isn't boomer babble it's just saying that some people got extremely lucky to be early adopters in areas that became desirable. People want to hop right into a highly desirable area and are shocked to find out that having thousands of people wanting to do the same thing drives the prices up. It is disingenuous to say if only I had the money I would've bought in X location while it was cheap because hindsight is 20/20. It was cheap for a reason at the time.
Even if we disallowed investors and only allowed primary residences there is always going to be someone willing to pay more money than you to live in the spot that you want if it is a coveted area.
This is not to undermine the supply issue of housing which is due to archaic zoning laws in most cities, which disallow multifamily units from being constructed. This probably won't decrease surrounding area prices but can greatly slow down the increase.