[solved] GSdx recordings are off-color

[pseudonym]

I confirmed that gsdx's coefficients and offsets are correct (bar rounding) for one of the two main YUV standards, try these to use the other and see if colour matches:

const GSVector4 ys(0.062f, 0.614f, 0.183f, 0.0f);
const GSVector4 us(0.439f / 2, -0.339f / 2, -0.101f / 2, 0.0f);
const GSVector4 vs(-0.040f / 2, -0.399f / 2, 0.439f / 2, 0.0f);

[patrickdinh]

try these to use the other and see if colour matches:

const GSVector4 ys(0.062f, 0.614f, 0.183f, 0.0f);
const GSVector4 us(0.439f / 2, -0.339f / 2, -0.101f / 2, 0.0f);
const GSVector4 vs(-0.040f / 2, -0.399f / 2, 0.439f / 2, 0.0f);

Compiled GSdx with those coefficients you gave... the colors still don't look right in YUY2 output.

--------------------------------------------------

framegrabs from my recordings...

first image: using "current" (in SVN) coefficients
   

second image: using the provided coefficients (yellows looks better)
   

third image: "good" RGB screen
   

[pseudonym]

...well all I can think now is that it's a codec bug (which codec are you using to test?). There's mention on the internet of some codecs incorrectly using the full [0,255] range instead of [16,235] for Y and [16,240] for U and V (of course doing this internally is fine but the exposed interface must be standard) and histograms seem to show this. Of course if this is the cause it's not a gsdx bug at all and shouldn't be "fixed", but preferring RGB is a good idea regardless. These numbers are for [0,255] YUV:

const GSVector4 ys(0.114f, 0.587f, 0.299f, 0.0f);
const GSVector4 us(0.5 / 2, -0.331f / 2, -0.169f / 2, 0.0f);
const GSVector4 vs(-0.081f / 2, -0.419f / 2, 0.5f / 2, 0.0f);
const GSVector4 offset(0, 128, 0, 128);

[patrickdinh]

...well all I can think now is that it's a codec bug (which codec are you using to test?)

For codecs, I've tried uncompressed, Huffyuv, Lagarith, and ffdshow (libavcodec MPEG4). Recording in YUY2 resulted in the colors appearing incorrectly for ALL those codecs. For the above screenshots, I used Uncompressed and Lagarith (it depends).

I also did some quick RGB->YUY2 encodes of a recording with VirtualDub... the results looked fine, so it doesn't seem to be a codec bug; tried it with all 4 codecs I mentioned.

...

Those new coefficients definitely increase the contrast... but the colors still don't record correctly.

screenshot: using full-range coefficients/offsets
   

[pseudonym]

Huh, that's weird. I'm not opposed to your workaround but I really want an explanation and nothing seems to fit.

[pseudonym]

Okay, it actually is the code. A standard integer conversion worked so there's something wrong with gabest's SSE, which I didn't inspect too closely because of GSVector obfuscation.

[pseudonym]

Okay, it's actually a very silly problem, U and V are reversed. I was mislead by an apparent luma difference. Fixed in SVN.

[patrickdinh]

Okay, it's actually a very silly problem, U and V are reversed. I was mislead by an apparent luma difference. Fixed in SVN.

Awesome! YUY2 recordings look good now.

[Master_DX]

Hmmm, I'm surprise you found the error considering you only got one eye.

On a serious note though, Do all of them work now or was it only YUY2 fixed?

[pseudonym]

Only YUY2 was broken. (This is supported by most codecs though so pretty much any recording you made would be broken.)