New - ISO compression - help testing
#21
(04-27-2014, 11:02 PM)willkuer Wrote: I was just clarifying because linuzapps is doing that on-the-fly or? Do you want to include that?

I'm not sure what you're asking.

The linuzappz plugin doesn't do anything on the fly IIRC. It can compress an iso to a proprietary format which it can then read, but it doesn't happen automatically.

This is a different system. You compress the files yourself to gz, and then pcsx2 can read and play them.
Reply

Sponsored links

#22
Ah sorry... was some time ago I used it. I thought it compressed the iso during the first start-up. But there was just this button In the plugin-menu as I now see. Sorry for confusing.
Reply
#23
@avih: Good job but is the .tmp extension really a good idea?
Some cleaning programs (e.g. CCleaner) tend to search and delete files like .tmp or .bak and would cause some issues here.
Of course PCSX2 will just create a new one in this case but still wouldn't it be better to name them something like .index or .pindex instead ?
Reply
#24
(04-27-2014, 11:12 PM)Grove4L Wrote: @avih: Good job but is the .tmp extension really a good idea?
Some cleaning programs (e.g. CCleaner) tend to search and delete files like .tmp or .bak and would cause some issues here.
Of course PCSX2 will just create a new one in this case but still wouldn't it be better to name them something like .index or .pindex instead ?

We can change it if it proves to be a problem. The reason I used .tmp is because it's not _required_ and can be generated at any time. It just makes things faster if it's there. And when people want to free up HDD space, they will delete .tmp files - which is fine. They can be deleted.

Again, if it proves to be a problem, we can change the name to something else. But let's first see if it's a problem.
Reply
#25
(04-27-2014, 11:12 PM)Grove4L Wrote: @avih: Good job but is the .tmp extension really a good idea?
Some cleaning programs (e.g. CCleaner) tend to search and delete files like .tmp or .bak and would cause some issues here.
Of course PCSX2 will just create a new one in this case but still wouldn't it be better to name them something like .index or .pindex instead ?

Well so does firefox and likely other browsers when downloading files, i don't see no issues there Tongue
Reply
#26
It's a problem avih Smile
Reply
#27
Don't guess! Wink

Unrelated, I've been experimenting with the iso-gzip stuff lately.

The current code (in git or the exe attached at the first post) may occasionally decompress at 8M chunks, which is not fun. I got this solved locally already by making the access code more efficient, with zero penalty. I'll push it to git soon.

I also managed to decompress in a different thread, but decompressing 4M chunks can still take longer than I'd like (which may include the OS HDD seek time as well, though the standard non-gz iso reader may also suffers from this). So even with a thread games may still wait for the data. So threading improves several cases but doesn't solve everything. I'll probably push the threading code to git at some stage.

While on the subject of disk access in a thread, prefetching is also possible, such that if PCSX2 "sees" that the game disk access pattern is such that soon it will need to decompress more data, it will start to decompress that data in a thread before the game actually asks for it. But for this to work, the "guessing" that it will need to extract more soon should be pretty good, otherwise, it may extract stuff which the game will never actually request, resulting in more cpu load than required. I still didn't experiment with prefetching.

I also got an idea to decompress in much smaller chunks (256k or 512k or even 128K or smaller) such that each decompression is practically instant and the game will not notice the delay.

The basic problem here is that the index is optimized for quick access in specific intervals, and it's optimal to extract data between these intervals (which right now are set to 4M). So every access of even 2k right now requires extraction of 4M (though all the following requests within this 4M chunk will be cached already and therefore will not require further extraction and will happen instantly). And 4M can still be slower than the game is happy to wait for.

Making these intervals smaller will allow access in higher resolution and smaller intervals, and each extraction would be faster. But an index optimized for smaller access intervals is also bigger. So if I'd set the intervals to 256K, then the index size would be 16x bigger (compared to 4M intervals), but the extraction speed of each chunk would also be 16x faster so the game will probably not notice any delay at all. But 16x bigger index is very likely to be too big for anyone to like (and negate most of the compression space saving).

So my idea here is to keep the intervals big (and the index size small), but once a game requests a chunk (e.g. of 2.5K which is roughly a DVD sector), instead of extracting 4M, it'll extract e.g. 256K BUT it'll also remember it's internal state such that IF the next access request is for the chunk right after the extracted 256K, it would extract as fast as if the index had 256K intervals. It's like storing another tiny index which is only useful for direct access to the next chunk of 256K. But the good thing is that most of the time, this is exactly the extraction which we'll need.

So, I didn't try it yet, but I think it's possible.

Also, if anyone wonders if it's worth the time, then the answer is that it's fun to solve these problems, so yes for me.
Reply
#28
avih for the problem with index.tmp,why not change with .axl ? in this case all cleaning program don't recognize the extention and don't delete it Smile
Reply
#29
If the new complexity is stable, then why not Tongue2
Congtrats on the threading, too. As core counts increase this makes it practically free on the CPU side Smile
Reply
#30
(04-28-2014, 05:36 PM)avih Wrote: The current code (in git or the exe attached at the first post) may occasionally decompress at 8M chunks...

So my idea here is to keep the intervals big (and the index size small), but once a game requests a chunk (e.g. of 2.5K which is roughly a DVD sector), instead of extracting 4M, it'll extract e.g. 256K BUT it'll also remember it's internal state ...

Done and done (the latter after I emailed Mark Adler - co author of zlib and he confirmed that it's possible). The new code is on git.

So this is already perfect now in many cases. Index intervals are still at 4M, but extraction happens in 256K chunks, and sequential access is optimal (i.e. very quick and with no redundant decompression). So this means that the game is likely to notice a delay only when it "seeks" to a different area of the disk.

Caching is still up to 200MB, but in chunks of 256K rather than 4M.

I guess that other than refining the current code, the next step is asynchronous in a thread and prefetching. But this is already really good right now.

The prints to the console now happen only if the extraction took more than 10ms (because there are many many extractions now which only take 1-2 ms, so it's too much spam to print all of them).
Reply




Users browsing this thread: 1 Guest(s)