glReadPixels – OpenGL Framebuffer Antics – ShiVa Engine

glReadPixels – OpenGL Framebuffer Antics

ShiVa offers a number of rendering backends, depending on the hardware and operating system you want to run your game on. On Desktop systems, you will most likely be rendering in OpenGL, which also means that you have easy access to the framebuffer using C/C++ and standard OpenGL APIs. This example will show you how to extract color and brightness information from the framebuffer using glReadPixels() in order to implement a fake HDR/automatic gamma shift effect.

Polling the framebuffer

Unlike VR or many post processing effects, framebuffer analysis does not require you to switch ShiVa to offscreen rendering mode. glReadPixels will work either way.
You can easily hook into the framebuffer from two points. Every plugin exposes engine events to your own plugin code, the relevant code can be found at the end of the Plugin.cpp file:

void OffscreenImageAnalyzer::OnEngineEvent ( S3DX::uint32 _iEventCode, S3DX::uint32 _iArgumentCount, S3DX::AIVariable *_pArguments )
{
    switch ( _iEventCode )
    {
    case eEngineEventApplicationStart  : break ;
    case eEngineEventApplicationPause  : break ;
    case eEngineEventApplicationResume : break ;
    case eEngineEventApplicationStop   : break ;
    case eEngineEventFrameUpdateBegin  : break ;
    case eEngineEventFrameUpdateEnd    : break ;
    case eEngineEventFrameRenderBegin  : break ;
    case eEngineEventFrameRenderEnd    : break ;
    }
}

To get the “freshest” possible frame, you would call your framebuffer function directly in case eEngineEventFrameRenderEnd. However if you don’t care about latency of one additional frame, you can make your life easier and call your framebuffer function from onEnterFrame() inside a ShiVa AIModel – which is what we will do for the rest of the tutorial. A one frame delay is totally acceptable for our example effect, unlike other applications like VR.

glReadPixels()

The function glReadPixels() returns the color value of a given pixel in the framebuffer, starting with pixel 1,1 in the lower left corner of the rendered image.

void glReadPixels( GLint x, GLint y, GLsizei width, GLsizei height,GLenum format, GLenum type, GLvoid * data);

x and y being your coordinates, width and height the pixel rectangle (=sample) size. Both GLenums describe properties of the pixel data, and * data is a pointer to the return array itself.
glReadPixels is comparatively slow. It blocks the pipeline and uses the CPU instead of GPU. With over 2 million pixels to analyze per frame in a FullHD image, your game will almost certainly come to a crawl if you checked every single pixel. A small sample size of pixels from positions all over the frame will most likely suffice. In my tests, a sample size of 10 was already enough to produce good results, and sample sizes up to a couple of hundred points posed no performance problem. As soon as you are dealing with tens of thousands of points however, you will probably experience slowdowns.

Plugin implementation

We will poll the framebuffer with an onEnterFrame() plugin function call:

-- y: brightness
-- r: red
-- g: green
-- b: blue
local y,r,g,b = oia.getHDRlastBrightnessAndColor ( )

The C++ function calls glReadPixels() in a nested for-loop with predefined stepping. The returned color values in the pixel[] array used to compute the brightness based on an industry standard function and then averaged.

#include 
// don't forget to link opengl32.dll too!
int Callback_oia_getHDRlastBrightnessAndColor ( int _iInCount, const S3DX::AIVariable *_pIn, S3DX::AIVariable *_pOut )
{
    S3DX_API_PROFILING_START( "oia.getHDRlastBrightnessAndColor" ) ;
    // Input Parameters
    int iInputCount = 0 ;
    // Output Parameters
    S3DX::AIVariable nBrightness ;
    S3DX::AIVariable nRed ;
    S3DX::AIVariable nGreen ;
    S3DX::AIVariable nBlue ;
	if (_bInitialized && _bHDR) {
		// query current game resolution
		int w = S3DX::application.getCurrentUserViewportWidth();
		int h = S3DX::application.getCurrentUserViewportHeight();
		// initialize output varables
		double cy, cr, cg, cb = 0.0;
		unsigned int iterations = 0;
		// traversing all pixels with big stepping for performance reasons
		for (int ih = 1; ih < h; ih += _optionLineStepping) {
			for (int iw = 1; iw < w; iw += _optionPixelStepping) {
				unsigned char pixel[4];
				glReadPixels(iw, ih, 1, 1, GL_RGB, GL_UNSIGNED_BYTE, pixel);
				auto r = static_cast(pixel[0]);
				auto g = static_cast(pixel[1]);
				auto b = static_cast(pixel[2]);
				auto y = 0.2126*r + 0.7152*g + 0.0722*b; // brightness according to Photometric / digital ITU BT.709
				// alternative, cheaper and faster calculation:
				// auto y = (r + r + b + g + g + g) / 6;
				cy += y;
				cr += r;
				cg += g;
				cb += b;
				iterations++;
			}
		}
		// calculate average
		cy = cy / iterations;
		cr = cr / iterations;
		cg = cg / iterations;
		cb = cb / iterations;
		nBrightness.SetNumberValue(cy);
		nRed.SetNumberValue(cr);
		nGreen.SetNumberValue(cg);
		nBlue.SetNumberValue(cb);
	}
	else {
		nBrightness.SetNumberValue(0);
		nRed.SetNumberValue(0);
		nGreen.SetNumberValue(0);
		nBlue.SetNumberValue(0);
	}
    // Return output Parameters
    int iReturnCount = 0 ;
    _pOut[iReturnCount++] = nBrightness ;
    _pOut[iReturnCount++] = nRed ;
    _pOut[iReturnCount++] = nGreen ;
    _pOut[iReturnCount++] = nBlue ;
    S3DX_API_PROFILING_STOP( ) ;
    return iReturnCount;
}

RGB and Brightness evaluation

The returned values can be used in ShiVa in many ways. For instance, the color could be used to modify lights, material colors or HUD colors, but for this tutorial, we will use the brightness value to manipulate Gamma. Our goal is to brighten or dim the scene dynamically depending on the overall brightness, so you can see better in dark scenes while bright scenes don’t blind you anymore.

        if this._bGamma ( ) then
            local gc = ((255-y)/128)-0.4
            scene.setGammaCorrection ( application.getCurrentUserScene ( ), gc )
            hud.setLabelText ( hud.getComponent ( this.getUser ( ), "hdr.lcurrentGamma" ), "Gamma:" ..gc )
        else
            scene.setGammaCorrection ( application.getCurrentUserScene ( ), 1 )
            hud.setLabelText ( hud.getComponent ( this.getUser ( ), "hdr.lcurrentGamma" ), "Gamma off" )
        end

ShiVa’s gamma correction ranges from 0 to 2, with 1 being the default. Our returned brightness ranges from 0 to 255, so the conversion forumula is (255-y)/128. I am subtracting 0.4 from that value because the test scene in itself is quite dark and its maximum brightness never exceeds 130.

Video

Here is a short video illustrating how automatic gamma can improve the visibility in dark areas, simulating the adaptability of the human eye:

Outlook

If you were to adopt similar techniques into your game, you should definitely implement a buffer that smooths out brightness and color changes over time. Since the gamma correction function feeds itself and reacts to any changes it introduces itself, you can easily end up with an oscillating picture, which quickly gets brighter and darker from one frame to the next. A timed delay between changes which more accurately reflects the behaviour of the human eye is highly recommended as well.
You should also give a thought to the pattern/stepping in your sampling loop. Often, pixel 1,1 – or the lower quarter of the screen for that matter – is taken up by some GUI, which should probably not be factored into the scene brightness calculation. Most of the time, a small sample from around the center of the screen will be enough.
Gamma might not provide you with the look you want, since it tends to produce washed-out colors. Instead, you can also experiment with Contrast, Color Level,s Saturation or Bloom.
Your average brightness and color values will rarely be pure white/black/red/… so it is up to you to find offset values that match the mood of your scene.
Finally, doing image analysis on the CPU with a blocking/synchronizing call like glReadPixels is in itself a questionable approach. As long as you keep the sample size very small, you will get away with it.




Need more answers?

  • slackBanner