They could have tried that, but why? It's a long con that can have complications, especially for the Tyrells since Stannis was already married and had a queen.Why didn't they (Tyrells, Renly's people) just stand behind Stannis to get the Lannisters out of power and then murk Stannis right after. Or plant some foul shyt on Stannis to force him to the Nights Watch. Could've had him mysteriously die in the war to be more convincing. Let Renly be king and marry Margery. Considering they murked Joffrey off, it's weird they didn't think of this solution.
What they did actually worked. They had the love of the people. They had the food to rescue King's Landing from starvation (instead of letting Stannis have that honor). And they had the superior army and were going to easily crush him on the field. A victory against the military genius Stannis on the battlefiend would have made Renly look like a warrior/conqueror, not some third born that lucked into being king. They had the battle and glory won, they just couldn't account for a shadow demon.
Also, though Renly had no love for his brother I don't think he actually wanted to murder him in a sneaky way.