When climbing ladders, there's a strange smoothing effect with the camera.
If you stop on a ladder, you instantly cling to where you are on a ladder.
If you keep moving upwards while on a ladder, you slowly get smoothed to a point upwards.
A good way to test this, is to jump on a ladder, and keep moving.
After that, do it again but stop moving when you reached the ladder.