[BUG] Excessive drive activity with some codecs

[Rotareneg]

[Event]
When recording using the MagicYUV and Lagarith codecs the disk activity is much higher than expected for the actual bitrate of the recorded file, causing storage bottlenecks. This does not happen when using the Ut Video codec or the dxtory codec.

[Condition of an event]
When recording 1600x900 video at 60 FPS, the Ut Video RGB codec bitrate was about 140 MB/s,  the hard drive active time was about 50% and the disk queue length was about 0.5, and the recording worked without bottlenecks or any issues. When using MagicYUV or Lagarith to record RGB, Dxtory reported a storage bottleneck even though the bit rate was only 98 MB/s (84 for Lagarith.) The drive active time was almost 100% and the queue length was nearly 1.0 even though the video bitrate was well below the write speed of the drive.

When recording video in yuv420 format, Ut video recorded at 67 MB/s and had a drive active time of 25%. MagicYUV in 420 recorded at 66 MB/s but had a drive active time of 82%, which seems very strange as one codec outputting data at the same bitrate and giving the same size file as another should not be using more than twice the amount of drive bandwidth.

For reference, the dxtory codec in the same situation recording RGB without compression was writing at over 200 MB/s, so the drive speed is NOT the issue here.

[Environment Information]

Code Select Expand

[Dxtory]
Version: 2.0.136
UID: dc281c35-d676-4a8a-9fa6-399b369bc830

[Dxtory Files]
InstallPath: C:\Program Files (x86)\ExKode\Dxtory2.0
amd_ags.dll Size: 271,872 bytes Ver:  SHA1: 44ab4f74637e23ca586580c6d746974e86da98e6
AppNames.txt Size: 1,047 bytes SHA1: 40cd39ab435dbcb3d44eba72b737a05b9a14410b
AudioStreamSplitter.exe Size: 38,432 bytes Ver: 2.0.0.2 SHA1: 5195bb906a508f9e7175f3c9c7d62c6c76f8217c
AudioStreamSplitter_ja.txt Size: 134 bytes SHA1: 489a5264d73a47edcce7c92edb131cf63e5e5709
AVIFix.exe Size: 72,192 bytes Ver: 2.0.0.131 SHA1: 8743067b975002d014008cfb0db363dfb606c375
AVIFix_ja.txt Size: 288 bytes SHA1: 9cab20780686687b84f604c77c73aaae3fd8126d
AVIMux.exe Size: 166,432 bytes Ver: 2.0.0.131 SHA1: 434d85edc5d4bb77c64ca703f9790062cc87a117
AVIMuxCore.dll Size: 199,200 bytes Ver: 2.0.0.130 SHA1: 53661b27f6f6f9f3e8316dca872fd4b01da93e49
AVIMux_ja.txt Size: 1,701 bytes SHA1: fa240715d701f450bb95377aa8517a252329b619
CryptData.pup Size: 84 bytes SHA1: cf9d724807f965d372da4d168bebd78c5258d6ae
Dxtory.exe Size: 606,928 bytes Ver: 2.0.0.136 SHA1: 62c1e8ee36a334439b70898ff8c3cd03f3b86a21
Dxtory64.exe Size: 130,112 bytes Ver: 2.0.0.132 SHA1: 4cce60b896a04515767435c7fd29feefed3c1aac
DxtoryCore.dll Size: 8,341,160 bytes Ver: 2.0.0.136 SHA1: e7b7f96718b7eb236d7195016967df5a7725170d
DxtoryCore64.dll Size: 8,064,720 bytes Ver: 2.0.0.136 SHA1: 1a448b0411ca7330b0750b5e460f02ef1548bf7d
DxtoryHK.dll Size: 199,376 bytes Ver: 2.0.0.136 SHA1: c583b69dbb12d190cddf5f6b4b7bab103b095d41
DxtoryHK64.dll Size: 230,096 bytes Ver: 2.0.0.136 SHA1: cb1509d2913ee731ea9ae3914a36e2a6b6db29ef
DxtoryMM.dll Size: 1,125,952 bytes Ver: 2.0.0.132 SHA1: b0b6fcc47b6b3f326de582975b269d57c2dd0594
DxtoryMM64.dll Size: 1,139,776 bytes Ver: 2.0.0.132 SHA1: 4cc91746487d863e2eea7a572c498c7eddf83817
DxtoryVideo.dll Size: 435,408 bytes Ver: 2.0.0.136 SHA1: 1aadce1a111f7ef840057129a673d6075c1f0edf
DxtoryVideo64.dll Size: 480,464 bytes Ver: 2.0.0.136 SHA1: 36bc4cc08f82a5a605a77f2d58fbfcea1699e18e
DxtoryVideo64_Install.bat Size: 57 bytes SHA1: 5c00dcbc4b631932e7920e7e1bf69e1daf8e26fe
DxtoryVideo64_Uninstall.bat Size: 60 bytes SHA1: 97986d6e7a40eceb84609444ed621c6689526c4d
DxtoryVideoSetting.exe Size: 129,568 bytes Ver: 2.0.0.130 SHA1: ee440e846cfeec2acb4a88f81f589e00c34e825e
DxtoryVideoSetting_ja.txt Size: 1,675 bytes SHA1: e532f36bd5ac6255659d240180637484661f0c83
DxtoryVideo_Install.bat Size: 55 bytes SHA1: 8494f535a15ea10b2d5e0fd62290a88758bf210e
DxtoryVideo_Uninstall.bat Size: 58 bytes SHA1: 6d05923ef8db62f30ee22582f06979f4b2b48a64
DxtoryVulkanLayer.dll Size: 109,776 bytes Ver: 2.0.0.136 SHA1: b8550335c1367d808b2bc64183ecda4ca61927ee
DxtoryVulkanLayer.json Size: 385 bytes SHA1: 2416b04e3895149b4200c1777f470a768519642d
DxtoryVulkanLayer64.dll Size: 132,816 bytes Ver: 2.0.0.136 SHA1: e9c6eaa8d2ad7d684c9525787455d3aca38f6598
DxtoryVulkanLayer64.json Size: 387 bytes SHA1: 13d84a82bda5f98cb6ad6069c13c9ce2304b7d38
Dxtory_ja.txt Size: 19,555 bytes SHA1: 6556833be873a4b7648d78ae8fdc838fe05af11f
EULA_en.txt Size: 1,356 bytes SHA1: 340c6b3425ab78d83c6a8bd344b6cc58d152ce8f
EULA_ja.txt Size: 1,814 bytes SHA1: cded0d38ec6a1cb0e86bc264496deaa28082cb51
filelist.txt Size: 279 bytes SHA1: e064a7b8f768b095f1f83ff59173763428a4d4f9
filelist64.txt Size: 289 bytes SHA1: f8ad619e674235657753020568c817707f767037
ignore_module_list.txt Size: 1,328 bytes SHA1: a1a91bc0bfb15d9202d3160c31c3aba39cd27a4c
InstallVulkanLayer.bat Size: 69 bytes SHA1: 156439c9308c8b5a32d697e1f3c11c2c5199a64b
InstallVulkanLayer64.bat Size: 71 bytes SHA1: 35574f9f2a414e8ebc592a8cec5a83d030b07000
LicReg.dat Size: 12,645 bytes SHA1: e203c8647c5d6d28ebe96f4957815c81eb99795f
LicReg.exe Size: 95,952 bytes Ver: 2.0.0.136 SHA1: 3382997b62809ac29762a6ea62197c6eb4c24094
LicReg_ja.txt Size: 707 bytes SHA1: 981af64897276c22867952f4972d9c29c6ee35e7
RawCapConv.exe Size: 208,928 bytes Ver: 2.0.0.131 SHA1: 32b6ec5d2f6649df4a47aedc476965433a8bdec4
RawCapConv_ja.txt Size: 1,869 bytes SHA1: 12086976f07abfc8d435abeee6c450c3cb2b07cd
RCStream.dll Size: 164,352 bytes Ver: 1.0.0.76 SHA1: 31266452f142cd7c856384945ffc410b2c9c8be9
readme.txt Size: 1,322 bytes SHA1: f8b7e31de7927309f7f3e03936374837906a94d9
Src16x10_Dest4x3.png Size: 40,081 bytes SHA1: 3b4f6332b8b3129ee351a9c96a9c4ac7f6f7a4cd
Src16x9_Dest4x3.png Size: 40,179 bytes SHA1: fcec824f4e95f7becf65bbabcc1a99c1470b99a1
unins000.dat Size: 23,933 bytes SHA1: af5d5da889f18b2104cb0cf99abd9142166dc217
unins000.exe Size: 1,198,288 bytes Ver: 51.1052.0.0 SHA1: f575ea89423ec66888075a109efef887ed1e8e7f
unins000.msg Size: 22,701 bytes SHA1: ab7961d64f8b22503c9c7ca4b335fa539b0278a3
UninstallVulkanLayer.bat Size: 71 bytes SHA1: f54f1fabd821b9e82086eebae8eb89e4e9322f86
UninstallVulkanLayer64.bat Size: 73 bytes SHA1: 3646bdc62ab4d18d3e8bf7e261dbc03ee7d77bfc
UpdateChecker.exe Size: 93,696 bytes Ver: 2.0.0.0 SHA1: 221bd3663f4393b368ebf74d1a892f9379ddbe44
UpdateChecker.ini Size: 238 bytes SHA1: 796da23b4a505bf2ef370dc271d1a80585dff6e2
UpdateChecker_ja.txt Size: 190 bytes SHA1: 969913786a3497f3e448572f7810ee9814314928
UWPHelper.dll Size: 132,304 bytes Ver: 1.0.0.1 SHA1: 5e6de832fad6d5d40bba66e568abc780cd93f9b4
UWPHelper64.dll Size: 159,440 bytes Ver: 1.0.0.1 SHA1: 9ddfab3d06bb0894341ba63a8dab98c83222d38f
DxtoryCodec.dll 64bit Size: 2,606,144 bytes Ver: 2.0.0.132 SHA1: 7ef69696fe270d8bf353ead32bd58681ee61e5fe
DxtoryCodec.dll 32bit Size: 2,499,648 bytes Ver: 2.0.0.132 SHA1: cd610194e3ddceefa04bc0f28a5f2139fee0d615

[System Information]
CPU: Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz
CPU Instruction Set: MMX SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2
OS: Microsoft Windows 7 Professional  x64
OSVersion: 6.1.7601
OSLanguage: 1033
ServicePack: Service Pack 1
Memory0: BANK0 4,294,967,296 bytes
Memory1: BANK2 4,294,967,296 bytes
TotalMemory: 8,589,934,592 bytes
Total Physical Memory: 8,587,890,688 bytes
Available Physical Memory: 5,763,166,208 bytes
Total Page File: 17,173,876,736 bytes
Available Page File: 13,471,940,608 bytes
Total Virtual Memory: 2,147,352,576 bytes
Available Virtual Memory: 1,646,829,568 bytes
PerformanceCounterFrequency: 3190048 (0x30AD20)
VideoCard: NVIDIA GeForce GTX 760 Memory: 2,147,483,648 Processor: GeForce GTX 760 DriverVer: 21.21.13.7254 DriverDate: 2016-08-10 19-00-00
Display0: 1920x1080 32bits 60Hz
SoundDevice: Virtual Audio Cable Manufacturer: EuMus Design
SoundDevice: High Definition Audio Device Manufacturer: Microsoft
SoundDevice: ASUS Xonar DGX Audio Device Manufacturer: ASUSTeK
SoundDevice: NVIDIA Virtual Audio Device (Wave Extensible) (WDM) Manufacturer: NVIDIA

[NVIDIA Info]
NVDriver: 372.54 r372_53
GPU0: GeForce GTX 760 Memory: 2048MB MemoryClock: 3004MHz BaseClock: 1085MHz BoostClock: 1150MHz

[Audio Device]
Speakers (ASUS Xonar DGX Audio Device) - CoreAudioOut
Line 1 (Virtual Audio Cable) - CoreAudioOut
Line 2 (Virtual Audio Cable) - CoreAudioOut
S/PDIF Pass-through Device (ASUS Xonar DGX Audio Device) - CoreAudioOut
Line 3 (Virtual Audio Cable) - CoreAudioOut
Stereo Mix (ASUS Xonar DGX Audio Device) - CoreAudioIn, Volume 100
Wave (ASUS Xonar DGX Audio Device) - CoreAudioIn, Volume 100
Line 2 (Virtual Audio Cable) - CoreAudioIn, Volume 24
Line 1 (Virtual Audio Cable) - CoreAudioIn, Volume 24
Line In (ASUS Xonar DGX Audio Device) - CoreAudioIn, Volume 100
Line 3 (Virtual Audio Cable) - CoreAudioIn, Volume 24
Microphone (ASUS Xonar DGX Audio Device) - CoreAudioIn, Volume 100
Aux (ASUS Xonar DGX Audio Device) - CoreAudioIn, Volume 100

[Video Codec 64bit]
Fraps Video Decompressor FCC:fps1
Intel IYUV codec FCC:iyuv
Lagarith Lossless Codec FCC:lags
Logitech Video (I420) FCC:i420
MagicYUV lossless v1.2rev0 (64 bit) FCC:magy
Microsoft RLE FCC:mrle
Microsoft Video 1 FCC:msvc
UtVideo Pro YUV422 10bit VCM FCC:uqy2
UtVideo RGB VCM FCC:ulrg
UtVideo RGBA VCM FCC:ulra
UtVideo YUV420 BT.601 VCM FCC:uly0
UtVideo YUV420 BT.709 VCM FCC:ulh0
UtVideo YUV422 BT.601 VCM FCC:uly2
UtVideo YUV422 BT.709 VCM FCC:ulh2
x264vfw - H.264/MPEG-4 AVC codec FCC:x264

[Video Codec 32bit]
Cinepak Codec by Radius FCC:cvid
DebugMode FSVFWC (internal use) FCC:dfsc
Fraps Video Decompressor FCC:fps1
Intel IYUV codec FCC:iyuv
Lagarith Lossless Codec FCC:lags
Logitech Video (I420) FCC:i420
MagicYUV lossless v1.2rev0 (32 bit) FCC:magy
Microsoft RLE FCC:mrle
Microsoft Video 1 FCC:msvc
UtVideo Pro YUV422 10bit VCM FCC:uqy2
UtVideo RGB VCM FCC:ulrg
UtVideo RGBA VCM FCC:ulra
UtVideo YUV420 BT.601 VCM FCC:uly0
UtVideo YUV420 BT.709 VCM FCC:ulh0
UtVideo YUV422 BT.601 VCM FCC:uly2
UtVideo YUV422 BT.709 VCM FCC:ulh2
x264vfw - H.264/MPEG-4 AVC codec FCC:x264
Zipped Motion Block Video v0.1 FCC:zmbv

[Audio Codec]
Microsoft IMA ADPCM
Microsoft CCITT G.711
Microsoft GSM 6.10
MS-ADPCM
LAME MP3
MPEG Layer-3 Codec
PCM
IEEE Float

[License Information]
License: Registered
Expire: Unlimited
LicVerify: 0807ab912c13dc6e9c3ad6db3b7f8a478373594c

[Additional Information]
Dxtory_log.txt: Empty
ErrorLog.txt: Empty

[Global Setting]
OptionFlag=SharedJavaProfile

[Ignore List]
C:\Program Files\Sony\Movie Studio Platinum 13.0\MovieStudioPlatinum130.exe

======================================================================

[Profile]
Name: Default
Path:

[Overlay]
Position: LeftTop
Flag: VidoeFPS RecordFPS RecordState SSNotification  Time Size Rate Warn
Layout: Vert
Color: Sleep FF00FF00, Active FFFFBC00, Movie  FF00FF00, ScreenShot FF00FF00

[Folder]
x M:\Video_capture  Speed: 202MB/sec  Free: 1,511,391,170,560 Total: 2,000,396,742,656 Format: NTFS
SSFolder:

[HotKey]
MovieCapture: Alt + 0 (0x0008000B)
PushToTalk:  (0x00010000)
ScreenShotSingle: None
ScreenShotRepeat: None
ScreenShotAuto: None
ToggleStatus: None
MovieCaptureFrame: None
ScreenShotCaptureFrame: None

[Movie]
Clipping:
ClipMargin: 0,0,0,0
ClipSize: 0,0
Scaling: Percent 100% 0x0
Codec: Dxtory Video Codec, RGB
FrameRate: 60
Output: File AVI
Option: IncludeCursor IncludeOverlay

[Audio0]
Enabled: True
PushToTalk: False
Device: Speakers (ASUS Xonar DGX Audio Device)
Codec: PCM
Format: PCM 48000 Hz, 16 bit, Stereo

[ScreenShot]
Clipping:
ClipMargin: 0,0,0,0
ClipSize: 0,0
Scaling: Percent 100% 0x0
HighSpeedLimit: 0
RepeatInterval: 1000
Format: PNG
JpegQuality: 97
Option:

[Advanced]
DataProcessingOption:
HookOption:
ThreadCount: 1
ThreadMask: 0000000000000000
LimitedFPS: False
FPSLimit: 60
DelayHook: False
DelayHookTime: 5

======================================================================

[Profile]
Name: javaw.exe
Path: C:\Users\Jon\Desktop\Minecraft\runtime\jre-x64\1.8.0_25\bin\javaw.exe

[LastUsingInformation]
DateTime: 2016/09/10 11:18:25
Version: 2.0.136
64bit Process
DeviceType: GL
View0.Type=GL
View0.Version=4.5
View0.Size=1600x900
View0.HWND=0x00020440
View0.WindowTitle=Minecraft 16w36a
View0.ClassName=LWJGL

======================================================================

[Profile]
Name: Steel Beasts Pro PE
Path: C:\eSim Games\SB Pro PE\Release\SBProPE64cm.exe

[LastUsingInformation]
DateTime: 2016/09/09 19:42:26
Version: 2.0.136
64bit Process
DeviceType: DX9
View0.Type=DX9
View0.DevType=D3DDEVTYPE_HAL
View0.BhFlags=D3DCREATE_FPU_PRESERVE D3DCREATE_MULTITHREADED D3DCREATE_HARDWARE_VERTEXPROCESSING
View0.Format=D3DFMT_X8R8G8B8
View0.Size=1600x900
View0.HWND=0x000E0410
View0.WindowTitle=Steel Beasts Pro PE
View0.ClassName=D3D Window

======================================================================

[Profile]
Name: Untitled - Movie Studio Platinum 13.0
Path: C:\Program Files\Sony\Movie Studio Platinum 13.0\MovieStudioPlatinum130.exe

[LastUsingInformation]
DateTime: 2016/09/09 19:51:12
Version: 2.0.136
64bit Process
DeviceType: GL DX9
View0.Type=GL
View0.Version=4.5
View0.Size=1178x580
View0.HWND=0x000307E6
View0.WindowTitle=Video Preview Window
View0.ClassName=Vegas.Class.VideoPreview.Output

======================================================================

[Profile]
Name: Untitled - paint.net 4.0.10
Path: C:\Program Files\paint.net\PaintDotNet.exe

[LastUsingInformation]
DateTime: 2016/09/10 11:40:32
Version: 2.0.136
64bit Process
DeviceType: DX11
View0.Type=DX11
View0.CreateFlag=D3D11_CREATE_DEVICE_BGRA_SUPPORT
View0.FeatureLevel=D3D_FEATURE_LEVEL_11_0
View0.Format=DXGI_FORMAT_B8G8R8A8_UNORM
View0.Size=1920x17
View0.HWND=0x000503EE
View0.WindowTitle=
View0.ClassName=WindowsForms10.Window.8.app.0.141b42a_r12_ad1
View1.Type=DX11
View1.CreateFlag=D3D11_CREATE_DEVICE_BGRA_SUPPORT
View1.FeatureLevel=D3D_FEATURE_LEVEL_11_0
View1.Format=DXGI_FORMAT_B8G8R8A8_UNORM
View1.Size=17x933
View1.HWND=0x0003039C
View1.WindowTitle=
View1.ClassName=WindowsForms10.Window.8.app.0.141b42a_r12_ad1

======================================================================
VerifyHash: 759c61164e594fefb86bd76c027c8d4bd563c8f2

[SnizzleLive]

I had a similar problem some time ago when trying to switch from UT Video Codec to MagicYUV in order to improve my rendering times. After multiple tests I noticed that using another harddrive for recording fixed the problem to some degree. What seemed strage was that the old HDD with which I got performance issues when recording with the MagicYUV codec had an estimated writing average of close so 190 MB/s and still created a storage bottleneck with laggy gameplay and ingame footage whereas the new HDD with a writing average of aproximately 120 MB/s created no such bottleneck.

When discussing this problem with a friend of mine the only explanation we could come up with was a different strategy of the two codecs regarding HDD usage. Maybe MagicYUV distributed its data to sectors which already have developed some errors over time or something like that. Problem is, this theory is hard to put to the test so I just ran with it.

If you think this problem is caused by Dxtory really, then using a different program might fix the issue? So while writing this it occurs to me trying a different program to verify where the real problem lies would be a good start. I guess I'll try that and report back when I get results

edit: After testing a bit with other recording software it seems this really is a problem exclusive to Dxtory. I would therefore like to report the same problem as Rotareneg

[Rotareneg]

Here's two screenshots showing the difference between recording with utvideo:



And MagicYUV:



When recording utvideo, it was writing about 133 megabytes/sec with a drive activity of about 55%. With MagicYUV it was writing only 111 megabytes/sec, but with a drive activity of 64%.

Now that I've got both those screenshots up next to each other I can see something else odd: when recording utvideo Dxtory had very little read activity, about 700 bytes/sec, but with MagicYUV it was reading around 250000 bytes/sec, a significant difference.

[ClassifyLP]

That isn't exactly odd, that could actually show why the activity is different. Maybe MagicYUV's approach uses old data to compress further.

[Ignus]

It has nothing to do with using "old data" or "HDD sector errors" and other such nonsense.

The root cause has been discovered here: http://forum.doom9.org/showthread.php?p=1781966#post1781966

Basically, Dxtory probably writes data inefficiently to disk in the case the compressed size is not an even number. AVI frame chunk sizes must be a multiple of 2 bytes, so if the compressed data size ends up being an odd number, it has to pad it with a single byte. Dxtory probably issues a separate system call to write that single byte, but writing single bytes the OS first has to read back the whole sector, which is 4096 bytes (4K bytes) usually, modify the byte, then write it back again.
At 60 FPS this means 60*4K = 240Kbytes of read-back in the worst case, which verifies Rotareneg's measurement.

This problem manifests for MagicYUV, Lagarith, x264vfw as well.

The reason it doesn't manifest for UtVideo is that Ut always produces compressed frames with sizes of even number of bytes.

So Dxtory should be fixed, but I'll also modify MagicYUV to produce even sized compressed frames in the next release (I tried, and it makes the problem go away).

Greets,
I.

[ClassifyLP]

It has nothing to do with using "old data" or "HDD sector errors" and other such nonsense.

The root cause has been discovered here: http://forum.doom9.org/showthread.php?p=1781966#post1781966

Basically, Dxtory probably writes data inefficiently to disk in the case the compressed size is not an even number. AVI frame chunk sizes must be a multiple of 2 bytes, so if the compressed data size ends up being an odd number, it has to pad it with a single byte. Dxtory probably issues a separate system call to write that single byte, but writing single bytes the OS first has to read back the whole sector, which is 4096 bytes (4K bytes) usually, modify the byte, then write it back again.
At 60 FPS this means 60*4K = 240Kbytes of read-back in the worst case, which verifies Rotareneg's measurement.

This problem manifests for MagicYUV, Lagarith, x264vfw as well.

The reason it doesn't manifest for UtVideo is that Ut always produces compressed frames with sizes of even number of bytes.

So Dxtory should be fixed, but I'll also modify MagicYUV to produce even sized compressed frames in the next release (I tried, and it makes the problem go away).

Greets,
I.

That's great to hear, I will be able to get back to MagicYUV then.

[SnizzleLive]

(...), but I'll also modify MagicYUV to produce even sized compressed frames in the next release (I tried, and it makes the problem go away).

Thats great to hear! Thank you for the hard work, do you maybe have an ETA on the new version?
I'd hate to have stored up crazy amounts of data produced with UT codec instead of MagicYUV as they take way longer to encode

[Rotareneg]

Just in case anyone was wondering, this has not been fixed in Dxtory 2.0.139.

[Malix]

not on dxtory side, no. But the rev1 of magicyuv fixed it for me.

[Ignus]

Yes, 1.2rev1 is out which includes the workaround, so the disk activity issue seems to be fixed.
Big kudos to rotareneg for pointing in the right direction to nail down the issue!

Greets,
I.