I've had this problem in a number of games, usually the fps are 300-400 before I start recording without vsync, and when I start recording it records at 60, but there is terrible screen tearing.
The second I put vsync on, the games fps goes to 60(naturally) but dxtory struggles to get to 60, usually fluctuates between 20-55 causing unwatchable footage.
Most recently I'm having this problem with resident evil 7.
Under Advanced settings try enabling the setting "Wait for available buffer" when you are using Vsync in games. This sorted out my fluctuating recording fps with quite a few games.