[Wrong section] why not static recompiling?

[ignus879]

People who read the title will probably send me hate mail saying I shouldn't be a developer but a farmer, and I also know there is at least another similar thread.
I thing this can be an interesting reading for everyone. Pro who wants to answer and can spent some time here with me, and newbie who need to understand how emulator works.
However. Hi everyone, I'm Max, a newbie developer (maybe not so newbie in general, but I can't make any code different from C++ and Java, and i still have problems with triple pointers ).
I know that's not the developer section but I can't open new thread there (if someone can move this thread there it would me amazing ).
I'm introducing to the beautiful world of emulation.


I can't understand something, in particular the reason why static recompiler are off-limits due to the self modifing code.


But.
If self modifing code means I use lots of:
-address = 192
-JMP (address)
-...
-address ++
-JMP (address)

why can't I make this?:
1) create a virtual 'Ram' --> if PSX has 4Mb of ram (for example) I allocate 4 Mb on my windows computer.
2) i load the program in THAT part of the ram (operative system, game rom, whatever) starting at the first "dedicated" bit. Now the Windows Pc ram should be an exactly copy PSX ram.
3) Now "JMP (address)" shouldn't be a problem.

--> is the self-modifing code example right?
--> can i make what i describe in step 2?
--> is really the 'step 3 instruction' not a problem anymore?


And.
If it's not possible for reason 'xxx', can I make an hybrid recompiler that statically recompile for the Windows x86 client but has also a dynamic recompiler for JMPs?
So when you start the game, it loads the rom and a cache file, some part can be just execute because are recompiled, other are dynamicall recompiled.


And again.
Why use a static/dynamic recompiler to transform (eg) PSX assembly into x86 assembly? Is it difficoult to transform into an intermediate language (LLVM, C--, others | ok maybe not C-- but any sort of portable assembly just like LLVM) that can be later compiled for both x86 Windows and (eg) ARM Android?


I hope this was an interesting reading and thanks to anyone who will share his enormous knowledge .

-Max

[Blyss Sarania]

Did you call yourself a newbie developer? Dude, I have written lots of programs(including an incomplete 6502 emulator in basic), and gone to college for computer science and I just BARELY followed all that. Give yourself some credit!

[Leaper]

--> is the self-modifing code example right?

Not exactly, self modifying code would be something like following:

192: a; //some instruction a
194: address = 192;
196: (*address) = b; //replace instruction a with instruction b

What you've described is an indirect jump, which can be a pain to reproduce with static recompilation, but not impossible.

--> can i make what i describe in step 2?
--> is really the 'step 3 instruction' not a problem anymore?

No to both. The main problem with self modifying code and static recompilation is that translation is not necessarily one to one. Suppose example above is original assembly code you want to translate from say MIPS to x86. There is no equivalent for instruction 'a' in x86, so it has to be replaced with three instructions 'c1, c2, c3' and we got something as follows.

192: c1;
194: c2;
196: c3;
198: address = 192;
200: (*address) = b;

Now self modifying instruction at address 200 is incorrect. That instruction needs to be modified so that it actually modifies the x86 version of code. But how do you translate 'b'? What if 'b' is a dynamic value? What if 'b' translation takes more or less than 6 bytes?

And.
If it's not possible for reason 'xxx', can I make an hybrid recompiler that statically recompile for the Windows x86 client but has also a dynamic recompiler for JMPs?
So when you start the game, it loads the rom and a cache file, some part can be just execute because are recompiled, other are dynamicall recompiled.

I'm guessing this is more of a cost/benefit tradeoff. Compiled blocks are generally fairly small and need to be recompiled under many conditions, which makes them impractical to store and impractical to determine which of the cached blocks matches the current conditions. For blocks that do rarely change, recompiling them once in a long while does not really effect performance much, since recompilation is cheap enough.

And again.
Why use a static/dynamic recompiler to transform (eg) PSX assembly into x86 assembly? Is it difficoult to transform into an intermediate language (LLVM, C--, others | ok maybe not C-- but any sort of portable assembly just like LLVM) that can be later compiled for both x86 Windows and (eg) ARM Android?

Many projects do just that and it not any more difficult to translate to some Intermediate Representation than to translate to x86 directly. Doing the direct translation to x86, however allows for leaner x86 core and more control over performance. I'm guessing that's why PCSX2 went with the decesion to skip IR. Also none of the tooling you mentioned were available, when PCSX2 was originally started.

[refraction]

In answer to what i hope is an answer to some of your questions.

I don't think you can mix and match static/dynamic recompilation like that, at least not on the same unit, but as you stated code is modified all the time and some of the units pull data from seperate files rather than the main "executable", the VU's are a big hitter with this as they rely on data from the packed data files and are so constantly changing, some games have upwards of 80 different VU programs, where the ELF only runs on the EE side of things and is just one big program. We cannot staticly recompile the EE as that uses a lot of self modifying code, so it would be a major headache.

As for the virtual ram idea, we pretty much do that now. We used to have a system where we mirrored the layout completely of the PS2 memory map, but in order to do so we had to change a windows setting to allow it. Unfortunately this doesn't play well with many systems, so it wasn't an idea which was worth keeping.

As for the compiling to an intermediate language, this was done for speed. The demands of PS2 emulation are so high we need to squeeze every last bit of performance out of it that we can. Converting to an intermediate language will cause a huge speed loss, one we cannot go for. Maybe someone down the road will rewrite the recompilers to work for ARM, but for now, it will just be x86.

[ignus879]

Thank you everyone for the answers.

Now i know:
-LLVM traslation is possible
-static recompilation is not (for now)

So I tried to think something different:
Image this:

on the ipotetical console:
192: a; //some instruction a
194: address = 192;
196: (*address) = b; //replace instruction a with instruction b

on the new ipotetical Windows PC
192: c1;
194: c2;
196: c3;
198: address = 192;
200: (*address) = b;

trasformation:
192: a(); //like a function, it create a 'custom' istruction that makes c1, c2, c3
194: address = 192
196: (*address) = b //if 'b' doesn't exist, we create another 'function'

I know this can be slower, but my question is not about speed, but about static recompilation. I can't understand why it's so impossible.

Now I know we need 100% perfect copy of the ram to access variables and a 100% matching istruction set.

Have a perfect memor map is possible

As for the virtual ram idea, we pretty much do that now.

In my opinion it can be possible to have a copy of the istruction set
Try to think different: you ipotetically have twice the code of the program.
One line is the original code, the other is 'service' code

// start 'a' istruction
100: c1
101: NOP //no sevice istruction
102: c2
103: NOP //no sevice istruction
104: c3
105: NOP //no sevice istruction
106: JMP routine
....
383: routine = 384
384: JMP 100 // 'a' function, 384 = 192*2
385: NOP
386: address = 192*2
387: NOP
388: (*address) = b;

Only even istruction are the code of the program, the others are 'service' ones that can be used to implement a sort of virtual istruction set.
Each address is multiplied by 2 so it skips service istruction.

But.

How can i manage normal number from normal address? Is an intelligent idea multiply for 2 or no? Or it should just read the JMP address, multiply for 2, and then go to the line.

Is now this idea possible? Is the idea really completely impossible? Is almost impossible but a good developer can?

**When i write something like this i feel like Wright brothers designing the first airplane **

Thanks
Max

[refraction]

Theres a simple reason why its not possible. If the code self modifies it will be modifying ps2 assembley, thusthat modification will need to be recompiled again, which would make it dynamic recompilation, which is how it works now.

[ignus879]

Theres a simple reason why its not possible. If the code self modifies it will be modifying ps2 assembley, thusthat modification will need to be recompiled again, which would make it dynamic recompilation, which is how it works now.

I'm sorry
I didn't understand what you mean

If you have time to spend () try to quote where my code is not right, where i fail, which of my ideas are wrong.

PS: I'm not a so-smart-student, right?

[ignus879]

During this thread I learned a lot, especially that dynamic recompilation is the fastest solution we can have (NEVER DOUBT PCSX2 DEVELOPERS ).

But I still want to have a solution (a difficould, slow and bad one) to my problem about static recompilation.
I thought all the day and now I'll try to give a solution.

In this example i will use an ipotetical PSX console and an ipotetical Windows PC


My tecnique is divided in 3 parts:

1) create a 'virtual' set of istruction that lets you use all PSX assembly
eg: 'a' is a PSX istruction. I haven't the same istruction on PC but i have to translate it in 'c1' + 'c2' + 'c3' istruction.
In a separated part of ram I implement the istruction

// 'a' implementation
100: c1
101: c2
102: c3
103: JMP returnToCode

// custom 'jmp'
110: LOAD jumpVariable
111: MUL 3
112: STORE jumpVariable
--------// jumpVariable = jumpVariable *3
113: JMP jumpVariable


2) I allocate 3*Ram of the PSX. One will be the istruction of the virtual set, the others will be used as service for jumps

In another part of the ram, separated from the virtual set :

In PSX code:
192: a;
193: address = 192;
194: (*address) = b;
195: JMP address;

In PC code:
---------- istruction 1 ---------
200: returnToCode = 200+3 // this line + 3, 3 istruction is the step
201: JMP 100 // jump to make 'a' istruction
202: NOP // it has done 'a' operation in just 2 istruction, so the third is a NOP
---------- istruction 2 ---------
203: address = 192
204: NOP
205: NOP
---------- istruction 3 ---------
206: (*address) = b
207: NOP
208: NOP
---------- istruction 3 ---------
209: jumpVariable = address
210: JMP 110 //jump to 'custom jmp'
211: NOP

Ok i can eliminate the third NOP because it's always NOP. I thought I need at least 3 PC istruction to refer to the virtual set but I can do it in 2.

3) I already tell it, PC ram is the exacly copy of PSX ram (code is now *3 - or *2 times).


Now

What's wrong in my code? I'm sure that's something wrong. Or I really create the first static recompiler which support self modifing code?

Waiting for answers .
EDIT 1: I edited the code with the custom jmp

Max

[ignus879]

What's wrong in my code? I'm sure that's something wrong. Or I really create the first static recompiler which support self modifing code?

Happy halloween still waiting answers by someone who has time to spend especially by Leaper and refraction that seems to be very pro.

While I was waiting, i tried to make my idea easier to understand
So I create an .xls file (excel 1999-2003, not the new .xlsx format, for people that use Libre/Open/Super-duper/Office and not MS one)

Download it here (I can post download link, right? It's completely my creation upload to my personal Mega account, with CC copyright )

Reply me if I wrote something wrong (if possible also where and why )

Thank you for the time

Max

[Leaper]

With self modifying code it is usually hard to know what the value of the modification will be (i.e. you don't know statically what 'b' is). That value can come from external sources (e.g. files), or it can come through some complicated code sequence, such that you cannot trace it back, or it can take on many values based on other runtime values. Hence for your approach to work you would need to wrap every memory write in some code to correctly replace 'a' with 'b'. Also not all memory writes are modifying code, so you would need to be able to figure out, when write modifies code or data.

Anyways you approach is moving towards basically embedding an interpreter inside of the executable, with all the shortcomings of interpretation (speed, etc) and way messier implementation, than just interpreter + original code pair.