Everything I listed above still applies.
After all, the differential between what a player usually shoots and what he shoots when being defended by a particular player is all influenced by one another and the players that defended that player previously - what, when, how, where and who influenced that player's shooting %s up to the point of each game. It's basically a causality dilemma. You have the issue of someone who's an incredibly gifted scorer who shoots poorly from the field/or from a particular area (less than 6ft, less than 10ft, greater than 15ft - I believe are the DFG% areas) like Paul George whose percentages aren't always indicative of how well a defender guards him but more so with his shot selection, and he'll have the odd game where he can't miss no matter what shot he takes. You have the issue of players getting hot or getting cold for no reason centered around their defensive matchup. The issue of a defender's style and movements being more suited to certain offensive players. The issue of a defender not wanting to get into further foul trouble and playing more conservatively. The issue of players settling for outside shots against players more than they'd like to, if there's someone like Whiteside or Gobert deterring them from driving into the paint.
Again, the list is endless.