..:: PCSX2 Forums ::..

Full Version: New - ISO compression - help testing
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Do you have tests to verify this, avih?
It would be nice to see how tweaks like this affect the gameplay experience, and not only for compressed images but regular reads as well.
I wonder if there's some system monitor application that can track file accesses together with cache hits / misses.
(02-22-2015, 05:21 PM)rama Wrote: [ -> ]Do you have tests to verify this, avih?
It would be nice to see how tweaks like this affect the gameplay experience, and not only for compressed images but regular reads as well.
I wonder if there's some system monitor application that can track file accesses together with cache hits / misses.

It doesn't affect normal reads at all - it only happens for compressed files.

The normal reads already use similar system, but with the compressed files it was harder to implement, so until now it didn't have it.

The current implementation for compressed files is sort of a hack, but a very reasonable one. We might still improve it in the future, but from my tests (which are tedious since many tests require reboots to make sure the OS cache is clean) it shows considerable improvement.

Also, while the tests showed clear improvement when the chunks were not at the OS cache yet, they didn't show performance regression when the chunks were already cached. So it's a win-win.

Also, the prefetch is asynchronous and quite tiny anyway - 256k and only happen after we actually extract a chunk from the compressed file (which is not very frequent since the internal cache already has the data most of the time anyway - so we don't need to actually access the disk and extract it).

It's easy to test how the patch affects cases where the data is already cached by the OS: just run the game with the compressed iso with and without the patch and see if one performs better than the other.

But to test if it actually improves the case where the data is not cached by the OS, you'll most probably have to reboot before each try, and even then run the game only once, because by the second time you run it the data is already likely to be cached by the OS.
BTW, if anyone is interested to clean up the windows RAM cache without reboot (not useful for anything except for testing how stuff behaves after the cache is emptied), there's a utility called consume.exe which is part of the Windows Server 2003 Resource kit Tools.

Download here http://www.microsoft.com/en-us/download/...x?id=17657

Then open it with 7zip, then right click rktools.msi and "open inside", then find consume.exe and drag it to some folder to extract it.

To empty the cache, I used: consume.exe -physical-memory -time 10

It will run for 10 seconds and basically uses all the ram - which forces windows to clear all its RAM cache. Note that it will make most applications runs slow afterwards for a while until the cache is filled again with the stuff they use.

Running a game after clearing the cache is a pretty good simulation of running it after boot or after it's not been played for a while, such that the ISO data is not cached.
Does this really send windows ram to nirvana or just to the pagefile?
It uses the RAM, which forces windows to free up any cache it holds. It might cause running applications to store some of their ram at the pagefile. But they'll regain their ram soon afterwards.

It's a test tool. It's not a tool to "free up memory" even if it also does just that. The fact is that windows is quite good at managing its cache and storing useful stuff at RAM which is otherwise unused, so all those "ram boosters" or other tools which "free up ram" are quite useless in general and actually hurt performance most/all of the time until the caches are filled up again.

So if you want to use this, just close the big applications you have open first (web browser, etc), use the tool, fire up pcsx2 and test whatever you need, then re-open the big applications.
(02-22-2015, 05:12 PM)avih Wrote: [ -> ]Added a system which asks the OS (windows only) to prefetch the next chunk from disk after the current extract finishes. This should help with sequential access of fragmented files without any noticeable negative impact.

Since this prefetch is asynchronous, it doesn't slow down PCSX2.

This should help on cases where the disk chunk is accessed the first time before the OS had a chance cache it, e.g. when running a game which wasn't running for a while, but also on other cases where the OS just dumped its cache since the last time the game was played.

https://github.com/PCSX2/pcsx2/commit/6b48784

Glad to see you improving on this self-proclaimed hack of yours. I find it's a great addition to PCSX2, to the point where ALL of my images are now gzipped. Smile
Hey hey.
Anyone ever heard of LZHAM?

It's like LZMA (with similar compression) but widely optimized for decompression times.
https://github.com/richgel999/lzham_codec

There's a 7-zip plugin too
how does the CR compare to the new CSO compression ?
(05-20-2015, 06:41 PM)mirh Wrote: [ -> ]Hey hey.
Anyone ever heard of LZHAM?

It's like LZMA (with similar compression) but widely optimized for decompression times.
https://github.com/richgel999/lzham_codec

There's a 7-zip plugin too

I haven't heard of it till now, but other than better decompression speed than lzma (according to them), it still has the same issues.

While it's faster to decompress than lzma, it's still slower than zlib (including gz), and it's unoptimized inherently for random access the same way as lzma isn't.

This means either very slow random access, or a huge index to make it quicker - both options likely to make it end up overall worse than gzip.
Thanks for the pigz tip, it uses all my cores and compresses in around 30-60 sec per Iso
Now a tip for some of you that would like to Automate the process somewhat
1 Put All your Ps2 iso files in a folder, This folder should ONLY contain Ps2 iso`s
2 Put pigz.exe in same folder as Ps2 iso`s
3 create a bat file and paste the following into it (new text doccument with extention changed to bat)
Code:
for /r %%i in (*.iso) do (
pigz -9 "%%i")

4 double click bat to start, it will go through every iso in the folder and at maximin compression convert and delete the original iso. If you want to keep the original iso then chang the bat to this

Code:
for /r %%i in (*.iso) do (
pigz -k -9 "%%i")
Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16