I actaully took it as Joel being mad that Tommy basically has the life that Joel himself once had. Joel
Apparently at one time had a black wife, half-black daughter, and was an architect.
Tommy now has a black wife, about to have a half-black child, and the first thing we saw him doing this episode was BUILDING something.
Tommy went away and basically became Joel 2.0 while Joel was out roughing it with Tess.
I actually thought for a minute Joel was going to call him out on it when they were in the bar