Here's a quick diagram of what the functionality issue is. This is within Annotations on a standard viewport (although the same thing happens everywhere). The previous 2020 reference marker tools had independent control of the font size in the OIP. I understand that the system has been moved to a plugin style where you can fully customize the shape of the marker. But by tying the font size to the symbol itself, it requires a full rebuild of each marker when a different size tweak is required (to avoid the issue of random text sizes which IMO make any drawings look sloppy).
Additionally, creating our own "standards" which conform to a smaller standard size seems to be an issue because only certain markers can be scaled. The elevation marker for instance, does not have the ability to edit the size without breaking the ability to rotate the arrow in the OIP settings.