New - ISO compression - help testing
#1
This version of pcsx2 supports reading compressed ISO using gzip. This is still not at the git repository but hopefully will get there soon (so new git nightlies will not have this support for now).

(no, I want it too but 7z will probably never be supported, other compressions will maybe be supported, but not soon).

gzip is a very standard and well known format, and its compression ratio is similar to zip, and usually saves about twice space than NTFS compression (e.g. if NTFS compression compresses a file from 4G to 3.5G = saving 500M, gzip will probably save about twice which will result in about 3G). If you have a lot of ISO files, the space saving can quickly add up to many gigabytes.

Interestingly, while bzip2 should usually compress better than gzip, with the few ISOs which I tested, gzip compressed about the same as bzip2. 7z still usually beats most formats, but the 7z format is very hard/impossible to index efficiently (see next).

To compress an ISO with gzip, use 7z: right click the file -> 7z -> add to archive -> change "archive format" to gzip, and press enter (or use any other tool which can create gzip files). PCSX2 will also list *.gz files at the ISO browser dialog.

Compressed files are usually not "mounted" directly because accessing random points at the disk can be extremely slow due to the compression. To overcome this issue, PCSX2 will create an "index" which will allow it to access any part of the file quickly.

Creating this index can take a while (up to a minute for a 4.5G file), but PCSX2 will then save it to disk, such that the next time the gzipped iso is used, it will load the index and start playing instantly. The index size is about 0.7% of the UNcompressed file size (e.g. if the uncompressed ISO is 1G, the index will be about 7M). The console will show progress while indexing the file.

The index is saved at the same folder as the ISO, with extension .pcsx2.index.tmp. If the index is deleted then PCSX2 will create it again next time the game is booted from the gz iso.

[update] - the index file name was changed to [mygame.iso.gz].pindex.tmp - which is still the default. Since 2014-12-31 it also supports custom name and folder. See http://forums.pcsx2.net/Thread-New-ISO-c...#pid423709

rama and me tested it with few ISOs and bin (from cue/bin pair) compressed files, and it seems to be working well. It should also work with other supported ISO formats like nrg etc, but we didn't test it yet.

Please help testing, and especially report:

1. Are there bugs? i.e. things which work with normal ISOs but don't work with gz compression of the same iso?

2. Do you notice slowdowns at all? if yes, how much? does it make the game unplayable? is it too annoying?

Running a game from a compressed ISO does use more CPU when accessing the disk, but from our tests so far we couldn't notice meaningful slowdown. Please help us test it.

Note, this is only pcsx2.exe. Put it at the same directory as your current pcsx2.exe and run it. Also, it was build with VS2013, so you will need the same runtime as the svn nightly builds need (if you used a svn build recently, just drop it at the same folder and it should run without problems).

[update] - replaced the attachment (this one has .3 at the end, the previous one had .1). The previous version somehow expected w32pthreads.v4-dev.dll instead of w32pthreads.v4.dll. Other than that, they're identical.

Thanks.

[update] added new version (.5) which adds caching (50 chunks of 4M at most). Should solve almost all slowdowns, except for games which constantly access different places on the disk (notable example is Shadow of the colossus while traveling - which constantly stream stuff from all over the disk).

[update 2014-05] gzipped iso support is now stable and is part of PCSX2 (disabled obsolete attachments).

[update 2015-02] 7-zip doesn't use multiple threads to compress gzip, but pigz does and can compress much quicker. Thanks to karasuhebi for testing.

[update 2015-02] new prefetch system should speed up cases of fragmented HDD. See http://forums.pcsx2.net/Thread-New-ISO-c...#pid436433

[update 2015-08-20] hidden the old attachments again, I think the recent forum upgrade unhidden them accidentally.
Reply

Sponsored links

#2
Good idea....but Iso compression aren't already in one CDVD plugin?.......From my point of view,best choice are fixing performance issues and some little problems with the games (like widescreen patch) ....Of course i like much your work you made,but this kind of feature is like an optional in a car...Was cool but not essential.........

Btw: In the archive miss the dev version of W32Threadsv4...I can't try the program.....
Reply
#3
(04-20-2014, 03:23 AM)axlffx2forever Wrote: Good idea....but Iso compression aren't already in one CDVD plugin?.......From my point of view,best choice are fixing performance issues and some little problems with the games (like widescreen patch) ....Of course i like much your work you made,but this kind of feature is like an optional in a car...Was cool but not essential.........

Btw: In the archive miss the dev version of W32Threadsv4...I can't try the program.....

That's a bit complicated. For one side is like telling to the devs what they should do next. On the other side they are doing something they envisioned can be done just now. The problem of disk space is small one for some but a huge difference to someone else.

Well, performance issues is another delicate situation. How to measure it? It similar problem, to some is lacking disk space, and only really solved upgrading hardware.

As today most games run fast enough in midrange desktop although sadly only in well above average laptops. Some few games where it does not happens is not really a performance problem but more like "misbehaving" software code in the games. Kind of those games using some specialized native optimizations and workarounds (most of times to increase performance natively, as in gaining performance in PS2 itself).

I'm sure the devs will love to get some more detailed help in how to resolve these cases which might be fooling and bypassing all already existent PCSX2 optimizations to the point that, sometimes those optimizations may become a hindering factor even.

May look like is simple and could be resolved with simple patches but actually not so easily implemented at all, besides it should be something which does not harm the already achieved compatibility with other games.

There is no irony in that saying, any actual help and suggestion in this sense is welcome... but being so general as saying resolving compatibility and performance issues is not really clearing how this can be done.

PCSX2 is already so hard to emulate in LLE that doing it in HLE is the best hope for bellow minimal requirement machines. But then achieving high games compatibility will be a serious issue for HLE due the very "alien" architecture PS2 is compared with PC. It doesn't have even a clear separation on what is work for the CPU and work for the GPU.

The main current problem is due the way the flow is streamed in PS2 making hard to achieve the needed balance and timings between EE, VUs and GS. No sensible way to preemptively process those modules out of order in LLE. I can't really say it is even possible in HLE as well. And this can only make yet harder the already hard task of achieving high compatibility with games in HLE.

The point is that at this point one should not expect PCSX2 to pass for a deep structural change without losing it's characteristics and actually losing all that was already accomplished. It would be remaking it from scratch.

Besides the devs have not been idle in trying and increasing performance all the time. Actually just for keeping the performance level with the increase in accuracy is not a shabby feat at all.

Nowadays one can dedicate in trying making new emulators for SNES beginning with totally ignoring any performance tricks to dedicate completely on accuracy... at "it's time" some snes emulators had even part of their code made in assembly in almost despaired attempt in getting that needed last drop of performance. Time changes expectations.

Even if the nowadays focus in the hardware industry is in power consumption decrease it might lead to portable machines getting near the processing power of today above average gaming desktops. Which should resolve most of those "performance issues" moaned today. Was so for every older emulator, it is so today and possibly will be so for the future emulators for today's consoles.
Imagination is where we are truly real
Reply
#4
Game: Summoner 1
No slowdown, no glitches (yet). Will re-post of anything changes.
Hangs steady at 60fps and turbo (for those pesky loading screens and quest running around) runs about 119pfs.
I had to re-name my w32pthreads.v4 to w32threads.v4-dev.dll then copied the original back in to let my original svn work, does this void my test? Original burn made with Alcohol 52%
compressed from 1,203,808KB to 812,234KB.
Let me know if you need more info (or less). Rest of computer stats in profile.

PCSX2 1.3.0-0 - compiled on Apr 20 2014
Savestate version: 0x9a0a0000

Host Machine Init:
Operating System = Microsoft Windows 7 Service Pack 1 (build 7601), 64-bit
Physical RAM = 16320 MB
CPU name = Intel® Core™ i7-4770K CPU @ 3.50GHz
Vendor/Model = GenuineIntel (stepping 03)
CPU speed = 3.496 ghz (8 logical threads)
x86PType = Standard OEM
x86Flags = bfebfbff 7fdafbbf
x86EFlags = 2c100000

x86 Features Detected:
MMX.. SSE.. SSE2.. SSE3.. SSSE3.. SSE4.1.. SSE4.2.. AVX.. FMA

Reserving memory for recompilers...

Loading plugins...
Binding GS: F:\pcsx2-XXXX-windows-x86\pcsx2-5928-windows-x86\plugins\GSdx32-AVX-r5928.dll
Binding PAD: F:\pcsx2-XXXX-windows-x86\pcsx2-5928-windows-x86\plugins\LilyPad~2014-04-16.dll
Binding SPU2: F:\pcsx2-XXXX-windows-x86\pcsx2-5928-windows-x86\plugins\ZeroSPU2-r5822.dll
Binding CDVD: F:\pcsx2-XXXX-windows-x86\pcsx2-5928-windows-x86\plugins\cdvdGigaherz-r5822.dll
Binding USB: F:\pcsx2-XXXX-windows-x86\pcsx2-5928-windows-x86\plugins\USBnull-r5822.dll
Binding FW: F:\pcsx2-XXXX-windows-x86\pcsx2-5928-windows-x86\plugins\FWnull-r5822.dll
Binding DEV9: F:\pcsx2-XXXX-windows-x86\pcsx2-5928-windows-x86\plugins\DEV9null-r5822.dll
Plugins loaded successfully.

(GameDB) 9653 games on record (loaded in 138ms)
HotSwapping to new ISO src image!
HLE Notice: ELF does not have a path.


Initializing plugins...
Init GS
Init PAD
Init SPU2
Init CDVD
Init USB
Init FW
Init DEV9
Plugins initialized successfully.

Opening plugins...
Opening GS
Opening PAD
Opening SPU2
Opening CDVD
compressed can handle : 'F:\pcsx2-XXXX-windows-x86\isos\Summoner.mdf.gz'
compressed can handle : 'F:\pcsx2-XXXX-windows-x86\isos\Summoner.mdf.gz'
Index file not found. Scanning compressed file to generate a quick access...
50MB 100MB 150MB 200MB 250MB 300MB 350MB 400MB 450MB 500MB 550MB 600MB 650MB 700MB 750MB
---------> totin: 831727279, totout: 1232699392, index alloc: 9608056
-----------> index len is 293, file size is: 1232699392
---> free: index->list: 32792, index: 12
isoFile open ok: F:\pcsx2-XXXX-windows-x86\isos\Summoner.mdf.gz
Image type = DVD
* CDVD Disk Open: DVD, Single layer or unknown:
* * Track 1: Data (Mode 1) (601904 sectors)
Opening USB
Opening FW
Opening DEV9
McdSlot 0: F:\pcsx2-XXXX-windows-x86\pcsx2-5899-windows-x86\memcards\Mcd001.ps2
McdSlot 1: F:\pcsx2-XXXX-windows-x86\pcsx2-5899-windows-x86\memcards\Mcd002.ps2
Plugins opened successfully.
EE/iR5900-32 Recompiler Reset
Bios Found: USA v02.20(10/02/2006) Console
BIOS rom1 module not found, skipping...
BIOS rom2 module not found, skipping...
BIOS erom module not found, skipping...
(UpdateVSyncRate) Mode Changed to NTSC.
(UpdateVSyncRate) FPS Limit Changed : 59.94 fps
(SYSTEM.CNF) Detected PS2 Disc = cdrom0:\SLUS_200.74;1
(SYSTEM.CNF) Software version = 1.00
(SYSTEM.CNF) Disc region type = NTSC
ELF (cdrom0:\SLUS_200.74;1) Game CRC = 0x13E2774E, EntryPoint = 0x00100008
(SYSTEM.CNF) Detected PS2 Disc = cdrom0:\SLUS_200.74;1
(SYSTEM.CNF) Software version = 1.00
(SYSTEM.CNF) Disc region type = NTSC
Found Cheats file: '13E2774E.pnach'
Loaded 8 Cheats from '13E2774E.pnach'
Overall 8 Cheats loaded
Overall 0 Widescreen hacks loaded
Loading patch '13E2774E.pnach' from archive 'F:\pcsx2-XXXX-windows-x86\pcsx2-5928-windows-x86\cheats_ws.zip'
comment: Widescreen Hack
(Wide Screen Cheats DB) Patches Loaded: 1
Reply
#5
(04-20-2014, 08:32 AM)Dargon Wrote: ... I had to re-name my w32pthreads.v4 to w32threads.v4-dev.dll then copied the original back in to let my original svn work, does this void my test? ...

No, that's fine. Thanks for letting me know and for the tests. I've added a comment for that on the first post.

Still not sure why it happens, maybe something broke at our build system on git (the gz patch is based on top of git master rev a86f2615be9505da59b6ab58be77539de34f9b2d from 2014-04-17 which currently is the latest).

[edit] I suspect a visual studio glitch, but clean and rebuild all resulted in a "good" version which doesn't require the dev dll. weird. I updated the first post with the updated file. Thanks again for the info.
Reply
#6
Added new version (.5) which adds caching (50 chunks of 4M at most, vs only 1 chunk with the previous version). Should solve almost all slowdowns, except for games which constantly access different places on the disk (notable example is Shadow of the colossus while traveling - which constantly stream stuff from all over the disk).

Also, this version has a lots of debug prints to the console, mostly '@' for disc access when reading from the cache instead of from the disk, [-NN-] for switching (instantly) to a different cache block, and [**extracting**] when actually reading and decompressing data from the compressed ISO (the *extracting* part is the only thing which may cause slowdowns).

Updated the first post also with this new version.
Reply
#7
Now i testing the multi caching version with Persona 3 FES (for now works great....) One question for avih: The data decompressed from .gz goes on disk or RAM?
Reply
#8
The decompression is to RAM, and in chunks of 4M, where right now it keeps the last 50 used chunks in RAM (that's the cache). The only thing which is written to the disk is the index file - and only if it doesn't exist already (at the first time of booting from the gz image, or if it was manually deleted).
Reply
#9
Just curious(don't take it the wrong way).
What is the point of this if Linuzappz ISO already can compress the image and load it and it also automatically create the index.

Isn't it better to just use the code from Lunuzappz to support z\bz images without the plugin?
I just compressed an image with linuzappz and 7zip and the difference was 2mb less on the gzip archive
Reply
#10
(04-21-2014, 02:01 PM)avih Wrote: The decompression is to RAM, and in chunks of 4M, where right now it keeps the last 50 used chunks in RAM (that's the cache). The only thing which is written to the disk is the index file - and only if it doesn't exist already (at the first time of booting from the gz image, or if it was manually deleted).

Now i understand why works great on my Pc....I have 16 GB Ram @ 2400 Mhz ... Thanks for answer me Smile

(04-21-2014, 02:33 PM)vsub Wrote: Just curious(don't take it the wrong way).
What is the point of this if Linuzappz ISO already can compress the image and load it and it also automatically create the index.

Isn't it better to just use the code from Lunuzappz to support z\bz images without the plugin?
I just compressed an image with linuzappz and 7zip and the difference was 2mb less on the gzip archive

gz and bz are different,i compressed in gz format (with 7zip,with option Ultra mode),Persona 3 FES from 4.486.624 to 3.149.699 is really good Smile and performance (with high speed Ram) are perfect,like iso uncompressed.....
Reply




Users browsing this thread: 1 Guest(s)