A stencil buffer is an extra buffer, in addition to the color buffer and depth buffer (z-buffering) found on modern graphics hardware. The buffer is per pixel, and works on integer values, usually with a depth of one byte per pixel. The depth buffer and stencil buffer often share the same area in the RAM of the graphics hardware.
In the simplest case, the stencil buffer is used to limit the area of rendering (stenciling). More advanced usage of the stencil buffer makes use of the strong connection between the depth buffer and the stencil buffer in the rendering pipeline. For example, stencil values can be automatically increased/decreased for every pixel that fails or passes the depth test.
The simple combination of depth test and stencil modifiers make a vast number of effects possible (such as stencil shadow volumes, Two-Sided Stencil, compositing, decaling, dissolves, fades, swipes, silhouettes, outline drawing or highlighting of intersections between complex primitives) though they often require several rendering passes and, therefore, can put a heavy load on the graphics hardware.
The most typical application is still to add shadows to 3D applications. It is also used for planar reflections.
Other rendering techniques, such as portal rendering, use the stencil buffer in other ways; for example, it can be used to find the area of the screen obscured by a portal and re-render those pixels correctly.
glEnable(GL_STENCIL_TEST); // by default not enabled glStencilMask(stencilMask); // allow writing to stencil buffer, by default (0xFF) no mask. glClearStencil(clearStencilValue); // clear stencil value, by default = 0 glStencilFunc(func, ref, mask); // by default GL_ALWAYS, 0, 0xFF, always pass stencil test glStencilOp(fail,zfail,zpass); // by default GL_KEEP, GL_KEEP, GL_KEEP, dont change stencil buffer glClear(GL_STENCIL_BUFFER_BIT); // clear stencil buffer, fill with (clearStencilValue & stencilMask)
Test: ( ref & mask ) func (stencilValue & mask)
Depending on the three possible conditions of stencil function/depth function.
1. Stencil Test Function fails:
If say func is GL_NEVER, the stencil test will always fail. Neither Color/Depth buffers are modified. Stencil buffer is modified as per glStencilOp fail. If say glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP) then GL_REPLACE takes place and stencilValue = (ref & stencilMask) // will become ref
2. Stencil Test Function passes/Depth Test Function fails:
If say func is GL_ALWAYS, the stencil test will always pass, but depth test may fail. Neither Color/Depth buffer are modified. Stencil buffer is modified as per glStencilOp zfail. If say glStencilOp(GL_KEEP, GL_INCR, GL_KEEP) then GL_INCR takes place and stencilValue = (stencilValue+1) // will become 1
3. Stencil Function passes/Depth Function passes:
If say func is GL_ALWAYS, the stencil test will always pass. If depth test also passes. Both Color/Depth buffer are modified. Stencil buffer is modified as per glStencilOp zpass. If say, glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP) then Stencil values are not changed, only Color and Depth buffers are modified.
Typically Stencil buffer is initialized by setting depth buffer and color buffer masks to false. and then setting appropriate ref value to stencil buffer by failing the stencil test every time.
// disable color and depth buffers glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE); glDepthMask(GL_FALSE); glStencilFunc(GL_NEVER, 1, 0xFF); // never pass stencil test glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP); // replace stencil buffer values to ref=1 glStencilMask(0xFF); // stencil buffer free to write glClear(GL_STENCIL_BUFFER_BIT); // first clear stencil buffer by writing default stencil value (0) to all of stencil buffer. draw_stencil_shape(); // at stencil shape pixel locations in stencil buffer replace stencil buffer values to ref = 1
Now use the initialized stencil buffer and stencil test to write only in the locations where stencil value is 1
// enable color and depth buffers. glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); glDepthMask(GL_TRUE); // no more modifying of stencil buffer on stencil and depth pass. glStencilMask(0x00); // can also be achieved by glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); // stencil test: only pass stencil test at stencilValue == 1 (Assuming depth test would pass.) // and write actual content to depth and color buffer only at stencil shape locations. glStencilFunc(GL_EQUAL, 1, 0xFF); draw_actual_content();