@jeff prince I've finally been able to fix the issue !
Problem was coming from the source data... Even-though I thought I did a good job at having clean data input.
Previous workflow
Contour imported from surveyor
Polyline simplified (50mm deviation) and converted to 3D poly
Run 'Validate 3D data' to spot and fix eventual overlap
Create site model from source Data
Site model looks good and was fast to generate (good sign). Everything looks fine until issues with contour labels...
BUT after careful analysis of source data it appeared that some 3D poly vertex were not on the same elevation (100mm difference, very hard to see, specially with hundreds of contour lines). Thus imported surveyor 3D data were the problem.
Fix
Convert 3D poly to contour lines (select all 3D poly/Landmark/Create object from shape/Site modifier/Contour). Leave elevation to 0.
The faulty 3D poly will now appear as a contour (or pad) with a slope. select them all (Select similar Tool) and set slope to none.
Convert the contour lines back to 3D poly.
Now you have nice 3D poly with all vertex on same levels.
This will be my typical workflow for now on as 'The Validate 3D Data' only is not enough.
EDIT
Problems are back. Playing with site modifiers and grading limit back and forth are causing contour labels to go crazy. And once it's started nothing can fix it (to my knowledge).
Workaround
Keep a copy of your site model (e.g. another layer, turned off). Save you Site Model settings.
Once you are done with your Site Modifiers (and your contours labels are all messed up), use the 'original' copy to replace the messed up Site Model.
Apply your saved setting and it should be fine now. (The problems start to appear when you play with modifiers back and forth. Not when initially applied)