New - ISO compression - help testing

[axlffx2forever]

I compiled the program and tested on Persona 3 FES, Persona 4 and i noticed some improvements in emulation speed (with the hi-speed ram @ 2400 Mhz) respect normal .iso files! Great Work,Avih!!!!

Btw: in my version i have modified the size of memory cache (from 200 MB to 400) and changed index filename from .tmp to .axl

Edit: The new update here: https://github.com/PCSX2/pcsx2/commit/49...f1e1e96bd5 cause a strange problem in a line:


/* if list is empty, create it (start with eight points) */
if (index == NULL) {
index = (Access*)malloc(sizeof(struct access)); <--- THIS LINE
if (index == NULL) return NULL;
index->list = (Point*)malloc(sizeof(struct point) << 3);
if (index->list == NULL) {
free(index);
return NULL;
}
index->size = 8;
index->have = 0;
}



The red line cause this error: "impossible to assign a value of type "Access *" to an entity of type "access *"

The compiler permit me to make pcsx2.exe but when i start play the game,PCSX2 say in the log window: "(IsoFS) Invalid partition descriptor encountered at block 0x10: '' and

File not found.
Path: IsoFileSystem
(pxActionEvent) IsoFS could not find the root directory on the ISO image.(thread:EE Core)

the iso image compressed of Persona 3 FES yesterday worked well today,after the compiling,no.

[avih]

index = (Access*)malloc(sizeof(struct access)); <--- THIS LINE

Note that this line has been there since the very first version of gzip-iso, so it's unlikely that this line is what caused this revision to not work while the previous one did work.

It's possible that something doesn't work with this revision, though I compiled this revision with both VS2013 pro and VS2010 pro, and it works on both for me. I don't think it's that line you mentioned, but provide more details and hopefully we'll find the issue and fix it.

Please continue the discussion at the commit page here: https://github.com/PCSX2/pcsx2/commit/49...nt-6157482 , and provide as much details as possible, try the previous version again, etc.

Remember that you speak of two (probably) different problems:
1. The warning with the red line - _probably_ harmless.
2. That the latest revision doesn't read gz iso for you.

For that red line , try changing (Access*) to (struct access*) (notice the upper/lower case a), and same from Point* to struct point* if it also has a problem.

For the "not reading gz iso", try the change with m_zstate which I suggested (twice) at the commit page.

Please refer to both these issues when you provide info. Thanks.

Note that I don't claim that there's no problem, just that from your details so far I still can't narrow down anything.

And again, let's continue on the commit page, not here.

[rama]

Since I just ran into this problem... Please make sure that your build is using the very latest sources.
Use git pull -r on the master branch. I'll also attach my release build here for you to test.

[avih]

Since I just ran into this problem... Please make sure that your build is using the very latest sources.
Use git pull -r on the master branch. I'll also attach my release build here for you to test.

Just to clarify, rama tried to build an old version which didn't read the gz iso files (because it didn't have the gz support yet), but after he updated to the latest it works fine for him, and he also attached the exe which works, so you can test with it.

[rama]

I love Git! It's really quite fabulous

[GeoMan]

Probably not the right place for this question here, but without modification the PCSX2 Bonus frontend will not be able to use .gziped isos...

[avih]

Added another significant speedup on several cases and bug fix some rare cases. I think now it's as fast as it could be (with synchronous extractions) and the extraction is probably optimal (minimal possible). RAM usage is roughly up to cache size plus 2x index size (up from cache + index).

Cache size is unchanged and still at 200M. Index files name/format/configuration also unchanged (so previous index files are compatible).

I think there's nothing left to squeeze before adding async extraction (thread).

[mirh]

Profane question: did you considered .xz file format?

As far as I understand it has the simplicity of gzip, but with the compression ratio of LZMA
And lzo?

[avih]

Not profane at all, and yes I have, and we might still add it.

Using gzip as a first compressed format was a good choice because the format is mature and stable, and (working) code sample, especially for direct access indexing, were available and easily incorporated. It also has the benefit of being relatively old and without major variants, so memory requirements are low (which is super important for an efficient direct access index of a compressed stream - because the index is equivalent to the decompressor's memory dump at fixed intervals throughout the decompression).

So, being able to get all the above stuff working almost out of the box made it easy to integrate compression into pcsx2, and then identify and work on related subject (like the kind of caching which would be useful, decompressor state storage, asynchronous decompression, etc).

Once we have a "system" for compressed iso support, we can certainly add more formats, and xz definitely sounds like an interesting one.

The problem, however, is that xz - being modern and all, can be configured during compression in such a way which will make indexing impractical. The usual tradeoff is: to get smaller files - you have to compress bigger chunks and use more memory during compression and decompression (and of course, newer algorithms which make use of this bigger memory).

And since people will always configure their compression to yield the smallest files, it also makes indexing these files rather impractical and will end in tons of support requests. In order to be able to index it efficiently, compression parameters have to be chosen in such a way which will hurt compression, and while xz can still produce smaller files than gzip when using such parameters, it usually happens to end up not much smaller.

gzip, with its simplicity and relative lack of configuration and low memory requirements, allows all files to be indexed relatively efficiently.

So yeah, zx is possible, interesting and we might try it, but it's much less straight forward than gzip.

[mirh]

I had quickly read other xz documentation and I thought it was easier to random access it

Other intriguing chatter here

I hope I'm not slowing down your work