I was having an issue like a few others I've seen on here where upon setting Dxtory to directshow output, any game I attempted to capture would crash instantly. Here's what I did to fix it:
Downloaded Revo Uninstaller, allowed it to load, and used it to completely uninstall Dxotry.
Installed .NET Framework 4.5 (I don't know if it had anything to do with it)
Downloaded the latest version of Dxtory from their website and reinstalled.
Hope this helps.