If the Quality setting of your OpenGL rendering mode has already been set to High or Very High, but you still jagged or serrated edges on some of your geometry:
It may be caused by one of the following:
1) You objects are far from the 0,0,0 3D origin. To correct this, move the geometry back to 0,0,0 and re-render in OpenGL. You can deselect all objects, then use View > Zoom > Fit to Objects to see if there is other geometry far from the origin. (This can occur if you have imported geometry from another software package.)
More information about document origin can be found HERE.
2) You are at a very high zoom level. This can sometimes occur at zoom levels higher than 5,000%. If your objects need to be viewed at this size, it would be better to change the scale of your design layers rather than remain at a high zoom level.
Edited by JimW