Actually he didn't get large until a year before we're introduced to him, they were racking up bodies, they were charged with 10 murders and were only convicted on 3, which happened to be one of McNulty's cases and that's what started the investigation. Marlo was introduced in the same way, only he "evolved" thanks to Chris hiding the bodies in the row houses.
Avon's run was short like Marlo's ...Marlo only became "the man" because Avon got hit with the twenty for parole violation, thanks to Stringer snitching.