Mixed hedging doesn't work when using the plant tool. The idea with the plant tool is that it only places one kind of plant at the time, either in single mode or in a poly-vertex/edge or array mode. This is the nature of the tool.
To place a mixture of plants, you have the Landscape area tool. This is a powerful tool, where you can set up mixes of plants you use all the time (either for planting beds or, as in this case, hedges). This tool works the way you say that you want to work - you create a geometry, either directly with the tool or by converting existing geometry, and you then tell it what plants you want it to contain and it will then give you the numbers.
If you say that you want to plant a mixed hedge with staggered rows and you use 5 plants per linear meter (per row), what you have to do is calculate how wide that double row would be, create a landscape area in this width and populate it with the mixture of your choice.
In the example below, I've first created a hedge according to this specification (as a comparison), then drawn a polygon to the size you want the hedge and converted it to a landscape area, in which I have put in several different species. (You can also see that it would go 25 plants in a m2 if you used the 5/lm size).
This way you also avoid having the individual plants showing. You can use the plant line on the landscape area to make the edge 'wiggly', and use any fill you'd like. Here I've just used a simple gradient, but you could use anything your practice normally use for hedging. (If you'd prefer, you can create yo your own line style specific for hedging and use that instead of the plant line in the landscape area).
If it would help you, I'll gladly send over the standard widths for different hedge units that I've collected over the years. Then you only need to use the double line polygon tool, set to the width needed to create your hedge area and then convert it to a landscape area.
This is a very effective way to work when making a lot of hedges- you create the hedge mixture for the landscape area once, saving it and then apply it to all the areas you want it on. Once you have created one area with your mix of plants, you can then select all the other areas and use the eye dropper tool to fill them all in one go. After that, remember to save the Landscape area mix so you can use it in your next project.