I added dimensions to the annotation layer and everything works fine. However, when myself or my coworker open the file again some dimensions have moved and rotated themselves.
Normally we draw them in the design layer but for this file we have different sheets. Each sheet shows the same layer except that they are at different scales and it is rotated. Therefore, I added the dimensions on the annotation layer.