You really think switching these two makes us worse?
The only argument is the defending and even then Brown isn't that good defensively anyway and like you said, we have great defenders on the team already, you can hide someone on defence if they're giving you 32 points and 7 assists on the other end. Look at the FT% too. How many times you see the Celtics go to the line and brick FTs?
Plus he can actually run an offence. Dame at the point over Smart? No brainer.