Adventures in Shading

posted by Chris on 24 Aug 2012

When LÖVE 0.8 was released in April, there were two main changes that drew my eye. First, it did away with Power of Two Syndrome, to which my reaction was: oh thank you. At that point, we were embarking on testing Sought and while we hadn’t yet run into someone whose drivers had this issue, it had all the makings of a nasty bug that would take some care to work around. The second change was the addition of pixel effects. To quote the breathless wiki entry:

Potential uses for pixel effects include HDR/bloom, motion blur, grayscale/invert/sepia/any kind of color effect, reflection/refraction, distortions, and much more!

To which I thought: cool. As I’m sure you’re aware, our games invoke a low-fi, retro aesthetic. As an emulation junkie ever since I found out I could play Atari 2600 games on my Power Macintosh 5400 (rest in peace, emulation.net), I knew people had written code to simulate the artifacts of an old television set or arcade monitor, and I knew that these effects could add a lot to the atmosphere of our games.

I don’t think either Joel or I have been very concerned about the authenticity of our games — e.g. whether or not Where We Remain could have been a NES game (which it couldn’t, not with our color palette and screen resolution). But I do agree with Jason Rohrer’s argument that the retro aesthetic goes beyond flat, sharp blocks of color. I think there’s something interesting in the analog artifacts of actual, not-yet-perfected hardware married to the idealized and brutal digital.

First Time Out

So I tried my hand at a pixel effect, just to get a sense of how hard it would be. I failed miserably. Pixel effects are written in a language called GLSL — not to be confused with HLSL, which is the DirectX take on the concept — which is essentially a stripped-down C dialect with a stock of vector- and matrix-related functions. Well, not exactly. They’re actually written in a LÖVE-specific thin layer of syntactic icing on top of GLSL. For example, the type sampler2D is #defined to Texture, which makes it a bit clearer as to its meaning, I suppose. LÖVE also takes care of the boilerplate of setting up a GLSL fragment shader, which incidentally is just the tip of the iceberg of 3D graphics jargon you encounter when you start working with this stuff.

(In plain English, a fragment shader computes the color for a single pixel of a 3D polygon, given a texture and pixel position. Its counterpart is the vertex shader, which manipulates the vertices of a polygon. As far as I can tell, shaders don’t have anything to do with color per se; the term seems to refer more to the idea of shading in a rough sketch.)

Anyway, I don’t think this icing is all that helpful. Right now there is very little documentation out there on how to write a LÖVE pixel effect (guess why I’m writing this blog entry?), though there are a bunch of examples on the forums. On the other hand, there are oodles of material on the web that explain GLSL, and similarly large amounts of sample code. If you can’t translate between GLSL and LÖVE’s dialect, they won’t do you any good, so you end up learning GLSL anyhow. It’s sort of a moot point, as it’s perfectly fine to use the GLSL types instead of LÖVE’s, and if you really, really want to, you can disable the sugar entirely.

That was my next stop. I tried plugging in a shader I found on the web — I forget which one, but it crashed and burned with a cryptic syntax error. In retrospect, I’m not sure if I just copied and pasted incorrectly or the shader expected a different version of GLSL. It’s hard to interpret error messages when you have no idea what you’re doing.

So to sum up, pixel effects are not something you can create without knowing at least a little bit about 3D graphics, nor can you plug in existing GLSL fragment shaders on the web as-is and expect things to work magically.

And Now The Learning Can Begin

I finally had some time in the last two weeks to dive into learning PixelEffects. Here’s a link-primer for those, like me, who are coming to the subject with nearly nil pre-existing knowledge:

  1. What-When-How’s introduction to GLSL shaders, parts 1 and 2. This doesn’t even talk about writing shaders per se, just the language itself. I actually read these twice: when I was first starting out, then after I had written a few effects. The second read settled things in my mind. After I finished, I finally felt competent.
  2. Gaussian Blur Filter Shader. Don’t worry about the math used to compute the blur too much — this is just a super-short example of a shader that is reasonably easy to understand.
  3. The GIMP documentation on image convolution. Image convolution is a simple but powerful technique that maps well to a pixel effect. It boils down to taking the average of pixels surrounding a single one, but how you weight that average dramatically changes the end result. You can blur, sharpen, and detect edges with the same algorithm. This documentation page is super light on the math, and has some nice examples.
  4. OpenGL GLSL Manual Pages. Your basic function reference.

There are a few things I know now that I wish I had known back when I first started out. First, texture coordinates have nothing to do with pixels. They instead range from 0 (top or left corner of the texture) to 1 (bottom or right corner). To translate these to pixels — e.g. to find a pixel adjacent to the current one — you need to know the size of the texture and multiply accordingly. The accepted practice seems to be for external code to tell the shader this through a uniform (or extern in LÖVE land) variable.

Which brings up the point that if you want to do post-processing effects in your game, you need to use a LÖVE canvas to do so. A pixel effect normally acts on each image or shape you draw individually, but this causes problems if you need the pixels of separate images to interact with each other– when blurring, for example. Instead, you need to draw the entire screen in an offscreen canvas, then draw the contents of the canvas onscreen using your pixel effect.

The problem with this is that not all graphics drivers support canvases. But in my limited testing, those drivers that aren’t canvas-capable aren’t capable of pixel effects, either. So that moral of the story is that if you want to embrace all possible users, you can’t depend on a pixel effect to convey something essential to the player.

Finally, for loops are ridiculously slow. Really! I tried using a set of nested for loops to implement a 3×3 convolution matrix and it was astounding how slower it was than an unrolled loop. Because I am just a mortal, I am willing to accept it as just one of life’s little WTFs and work around it.

Going Back To Last Century

Once I had a handle on GLSL, My first thought was to try porting an existing retro effect to LÖVE. The most common monitor effect in emulators is Blargg’s NTSC filter, which actually goes to the trouble of converting the pixels into an analog signal, then simulating a television poorly interpreting that signal. It looks really good… but it’s licensed under the LGPL. Porting it to GLSL would count as a derivative work, which would introduce a slew of license considerations. I was hoping to include the shader with Zoetrope, which is zlib/pnglib licensed.

I think the LGPL is completely right for this kind of work, and obviously it’s Blargg’s moral right to decide how his or her work is licensed anyhow. So it was time to dive in for real.

I knew that writing something that literally simulated an analog signal would take a lot of effort, so I did some research and it seemed that simulating an old monitor had these effects:

Once you give up the idea of actually simulating analog signals and rabbit-ear televisions, there are no hard and fast rules to creating an effect. There are many possible takes on the concept and none can really be considered definitive; it’s all up to taste. And besides– after playing some old games with a critical eye to the NTSC filter, it really does reduce the original pixels into a muddy mess. There had to be a decent middle ground.

I tried a lot of different approaches. Some did well with dark scenes but not well with bright ones. Some worked better with text than others. After a couple days of experimentation, my eyes started to hurt in that peculiar way they do when I’ve spent a lot of time designing color schemes for web sites.

It led to some interesting, bizarre effects.

At one point I had turned my shader into an illegible, blurry, washed-out mess. I put it aside and wondered if I just didn’t have the chops for it after all, or that I would have to learn the ins and outs of NTSC in order to get something that actually looked good. But mostly I thought instead of coding. I think you have to give up on big problems for a little in order to solve them.

Inspiration returned a day or two later, when I decided to start playing with the different filters on the emulators I had on my computer. Kega Fusion displays a screenful of static when you first boot it up; a nod to the old days of channel 3. But, as crazy as it sounds, the static looked perfect. I couldn’t distinguish it from a TV screen. I tried Ecco, my favorite Genesis game, and the effect wasn’t perfect but still compelling.

Kega’s menus make it manifestly clear the post-processing effects it uses: it does a bilinear filter on the image to blur it, applies scanlines at the intensity you select, and allows you to adjust the brightness of the image to compensate. All of these things are easy enough to code, and when I finished, the results looked good to me.

For fun I added a noise filter to it, which — to bring us full circle — is not particularly authentic, but I think it’s atmospheric. Here’s the source code.

In Closing

I’m hoping to roll pixel effect support into the next release of Zoetrope. I was hoping to offer people a way to use them without having to roll up their sleeves and understand their inner workings, but it doesn’t seem possible for now. There are just too many details that need understanding.

But more than that, I’ve enjoyed dipping my toes into the world of graphics. It’s a complicated, weird world full of words that seem to carry one meaning at first glance (shader, convolve) but hold another when you really understand them. It’s fascinating that way.