Is it possible to write a programm that detects where the Freelancer.exe calls the D3DX routine (D3DXQuaternionRotationYawPitchRoll) for the camera?
If I understood you right, you would just have to add that normalization after this point.

Thank you for making this point clear to me anyways

I'm crossing my little fingers and toes that it does mean that

There is no real way to fix this. The important thing to understand here is that Freelancer uses floating points (and I think, D3D8 also only supports floating points for camera stuff, but I may be terribly wrong there) for everything related to space positions and the camera.
When you are at 1000k and 1.0 = 1m, you have a precision of around 6cm.

Now imagine your ship standing at position 0, 1000000.03, 0
Since floating points cannot hold 1000000.03, but only 1000000.0 and 1000000.063, your ship and/or the camera may jump 6cm all the time. You see your ship shaking.
Also, what is important to understand: At those precision errors you can no longer fly in a straight line, your ship will shake when flying (or lets say, your ship will fly in zig-zag-pattern). This is especially visible at extreme distances, like 10000k, if you would login with 2 chars and would spectate the other player flying around.

Hmm, that's interesting. We haven't noticed this effect at FWs, and we have stations 500km from the centre.

Speak to blackhole, i believe he said something about this a while ago when he was talking about something similar, dunno if hell remember but im PRETTY sure it was him i was speaking to about this issue some time ago, tho i could be wrong.