pcsx2bench - optimization and benchmark tool
#1
Following this suggestion of mine, I decided to have a go at the benchmark tool, so here's pcsx2bench version 0.1.

It's a small (and quick'n'dirty) Win32/.NET2.0 command-line application and it works with GSdx, ZeroGS and ZZ-Open-GL. It measures performance by reading the output window's title (fps, etc) and does a little number crunching to come up with what I call Usable-speed rating. Rating of 100 represents 100% of the runtime in acceptable speed (= within 3% of optimal speed by default). The rating only takes into account the slow sections of the gameplay, so the rating should not change with frame-limiter on/off (the slow parts remain the same).

The tool operates in subjective PS2-time such that the same gameplay section will be measured regardless of the emulation speed (20 or 100 fps will still measure the same gameplay section).

For the most consistent results, it should be invoked on a saved-state that plays unattended (e.g. cut-scene or a replay). Repeatedly invoking the tool on the same saved-state while modifying Pcsx2's settings allows to find the best settings for this gameplay section.

The tool is inherently inaccurate due to the method by which it collects the performance data (reading another window's title? wtf?), but nevertheless, it's surprisingly consistent (usually around 1% or less) and usable. I hope that if the tool proves useful, the functionality will be replicated within Pcsx2 itself for much more accurate results.

For basic usage:
1. Run Pcsx2 and load a specific game you'd like to test.
2. Make sure F3 loads an existing saved-state, preferably of a cut-scene or a replay.
3. While Pcsx2 is running, run pcsx2bench.
4. Wait for about 20 seconds while it measures the performance and calculates overall rating.

If this tool proves popular, I'll probably release it under the GPL later.

Here's the usage info provided when the tool is invoked with -help:
Code:
Usage: pcsx2bench [-help] [-nokey] [-nostate] [-wait<nn>] [-bench<nn>]
                  [-fps<dd>] [-slow<dd>] [-pause] [-exit] [-quiet]

-help:      Displays this help and info page.

-nokey:     Doesn't wait for any key when done (convinient when using from
                 command line, allows to execute from explorer without args).

-nostate:   Doesn't load PCSX2's state (doesn't send PCSX2 the 'F3' key.)

-wait<nn>:  Waits nn seconds after loading the state and before starting
                   measurements to allow PCSX2 to stabilize. Default is 4 seconds.

-bench<nn>: Measures performance for nn seconds in PS2-time. Default is 15.

-fps<dd>:   Sets target fps which is considered 100% speed. Default is 59.94
                   (for NTSC). Use 50 for PAL-Europe games, etc.

-slow<dd>:  Sets minimum acceptable fps to be considered 'near-perfect'.
                    Default is 97% of full-speed fps (e.g. 58.1 for NTSC).

-pause:     Pauses PCSX2 when done (sending 'ESC' to the GS window).
                 Note: GSdx currently supports it, ZeroGS and ZZ_Open_GL don't.

-exit:      Exits PCSX2 when done (sending 'ALT-F4' to the main window).

-quiet:     Only outputs a summary or an error (usefull for shell scripts).

Examples:
Benchmark 60 seconds of a PAL game: pcex2bench -bench60 -fps50
Start without loading a state and without waiting: pcsx2bench -nostate -wait0

pcsx2bench monitors PCSX2's performance via the GS window title, and
calculates an overall usable-speed rating. Before executing pcsx2bench,
PCSX2 should be running with a game loaded, and a save state should exist and
loadable using 'F3' (unless invoked with '-nostate').

When executing pcsx2bench, it issues a load-state command to the output window
(sending 'F3' key), waits for few seconds to let PCSX2 stabilize, and then
measures performance for a pre-determined duration OF PS2 TIME (i.e. it
would measure THE SAME gameplay section wheather it runs at 100 or 30 fps).
When done, it displays a summary and an overall usable-speed rating.

The rating is calculated as follows: % of PS2 runtime at acceptable speed
plus the relative speed of min-fps and average-fps during the slow runtime,
e.g. If slow was recorded on 45% of PS2 time, min-fps=20, avg-slow-fps=30,
and target fps=59.94, The rating will be (100-45) + 45*(30+20)/2/59.94 = 73.8

Maximum rating is 100 - when it never gets below acceptable speed.


Note: If it fails to control PCSX2, Try runnning pcsx2bench as administrator.

Enjoy, and please provide some feedback if you have any.


Update 2010-11-14
---------------------
pcsx2bench 0.1RC2 (Bugfix):
- Rating of 100 is now displayed properly (was showing as 'NaN'.. /facepalm)
- -nokey option was ignored if main Pcsx2 window isn't found.


Updated 2010-11-27
---------------------
pcsx2bench 0.1RC5
- New: Live display of average speed (%) and minimum FPS while benchmarking
- Better timing accuracy (improved handling of ms rounding errors)
- Moved average speed result to the bottom
- Added examples at the info page


Updated 2010-12-04:
---------------------
pcsx2bench 0.1RC7
- New: Added support for ZeroGS/ZZ-Open-GL (FPS data only) and recent svn builds of GSdx.
- New: Commands '-pause' and '-exit' to control PCSX2 when done (not 110% reliable, only GSdx supports '-pause')
- New: Command '-quiet' to only display a summary or an error (useful for usage from shell scripts)
- Mod: Hopefully more reliable keyboard commands to PCSX2.


Updated 2012-08-05:
---------------------
pcsx2bench 0.1RC9
- Fix: Should now compensate for different numeric formats by locales (e.g. "FPS: 60,00" vs "FPS: 60.00").
- NOTE: _Might_ break if FPS goes beyond 1000. If that happens, just limit the FPS.


Attached Files
.zip   pcsx2bench_0.1_RC1.zip (Size: 7,16 KB / Downloads: 444)
.zip   pcsx2bench_0.1RC2.zip (Size: 7,17 KB / Downloads: 320)
.zip   pcsx2bench_0.1RC5.zip (Size: 7,46 KB / Downloads: 328)
.zip   pcsx2bench_0.1RC7.zip (Size: 8,79 KB / Downloads: 629)
.zip   pcsx2bench_0.1RC9.zip (Size: 8,83 KB / Downloads: 325)
Reply

Sponsored links

#2
Need to try out in practice but sounds swell! Smile
<REDACTED>
Reply
#3
pcsx2bench 0.1RC2 (Bugfix):

- Rating of 100 is now displayed properly (was showing as 'NaN'.. /facepalm)
- -nokey option was ignored if main Pcsx2 window isn't found.


First post updated with the new download.

Reply
#4
Has anyone tried it? working? useful? useless? good? bad? comments? suggestions?
Reply
#5
Hey avih,
I tested the app quickly yesterday and it works fine at what it does.
I personally don't think it's very useful as it is right now, mainly
because I know I can just turn on fraps and check the FPS it gives
me for a scene after loading a savestate.
Fraps rounds the FPS nicely, so I can notice even the slightest FPS differences.

Your tool could be used to benchmark more tricky things,
such as recompiler performance on level loads or similar.
I'll see if a situation comes up where I can try this.

Other than that, it's nicely bug free, I noticed Tongue2
Reply
#6
I have no idea how can you read reults.
Reply
#7
(11-15-2010, 03:45 PM)rama Wrote: Hey avih,
I tested the app quickly yesterday and it works fine at what it does.
...
Thanks Smile I don't use Fraps so I don't know the nature of the info it produces. Does it also report how much of the playback was at good speed, etc? Also, since this tool can measure the same gameplay section regardless of emulation speed, I thought it might be more accurate than tools which don't have a 'sense' of subjective PS2 time.

Anyway, thx for the feedback.


(11-15-2010, 04:00 PM)Game Wrote: I have no idea how can you read reults.
The final result, i.e. 'pcsx2bench rating' at the bottom is a metric I invented that tries to measure how good the game can be played in practice (speed-wise, regardless of visual glitches or inaccurate FPS due to excessive VU steal, etc).

The higher this rating is, the better the experience should be while playing, and the closer it can be to 100% speed at all times. Maximum rating of 100 is given if you can play it at 100% speed at all time of the measurement.

The rating is different than average speed, since even if it has an average of 70FPS, it can still sometimes get below 60fps. So this rating tries to take that into account too. It also never goes beyond 100 because when you play, you want to play at 100% speed and it's not really important how fast beyond that it can go. A rating of 100 says this entire measured section can be played at 100% speed.

Bottom line is, that you can modify pcsx2's settings (e.g. speed hacks, EE/VU advanced recompiler options, GSdx resolution, etc) and use this tool to give rating of how good the settings are in 'real-world play'. This takes into account not only the average speed, but also how bad it does when it can't reach full speed. So you want to find settings which gives you the highest rating on your system at this specific measured section.

The rest of the information is just some statistics that some people might be able to use. It includes overall speed, and statistics on the slow sections and the good sections, which includes fps, GS and EE load.

Reply
#8
Updated 2010-11-27
---------------------
pcsx2bench 0.1RC5
- New: Live display of average speed (%) and minimum FPS while benchmarking
- Better timing accuracy (improved handling of ms rounding errors)
- Moved average speed result to the bottom
- Added examples at the info page

* First post updated with latest download.
Reply
#9
Updated 2010-12-04:
---------------------
pcsx2bench 0.1RC7
- New: Added support for ZeroGS/ZZ-Open-GL (FPS data only) and recent svn builds of GSdx.
- New: Commands '-pause' and '-exit' to control PCSX2 when done (not 110% reliable, only GSdx supports '-pause')
- New: Command '-quiet' to only display a summary or an error (useful for usage from shell scripts)
- Mod: Hopefully more reliable keyboard commands to PCSX2.

* First post updated with latest download.
Reply
#10
As much as I would like it to be true(who wouldn't?;D), I guess it counts too many zeros:
   
(used RC7 without any additional commands, NTSC game, same problem with 1.0.0)
Reply




Users browsing this thread: 1 Guest(s)