Projection Mapping

Building Software

I was looking for a way to experiment with projection mapping without investing in expensive or complicated software. I came across a library for the Processing language called keystone, which got me on my way to initial experimentation. I am thankful to David Bouchard for the contribution to the community, however the documentation was a little light, and it took a bit of digging and messing around to be able to map multiple surfaces, customize the content and save the layout. So as a proponent of open source and giving back to the community I am including the code here for other people to develop and use. The code creates multiple mappable screens which can be loaded with video, images or things drawn in Processing. The information is then saved as a .xml file in the sketch folder, making it easier to load previous layouts of the mapping windows.

The Code

/*
*    Keystone Projection Mapping
     Requires the keystone library to function.
     http://keystonep5.sourceforge.net/

     Modified by Zoe Hardisty 2014
*/

import deadpixel.keystone.*;
import SimpleOpenNI.*;

SimpleOpenNI kinect;

Keystone ks;

int surfaceIndex = 0;
int surfaceCount = 100;
CornerPinSurface[] surfaces;

PGraphics[] textures;

int imageIndex = 0;
int imageCount = 10;
int currentImage = 0;
PImage[] images;

float aspectConst;
float scaleX;
float scaleY;
int offsetX;
int offsetY;

void setup() {
 size(displayWidth, displayHeight, OPENGL);
 kinect = new SimpleOpenNI(this);
 kinect.enableRGB();

 surfaces = new CornerPinSurface [surfaceCount];
 textures = new PGraphics [surfaceCount];
 images = new PImage [imageCount];
 ks = new Keystone(this);
 ks.toggleCalibration();
 
 // load the scale and offset settings
 loadSettings();
 
 addImage("1.png");
 addImage("2.png");
 addImage("blue.png");
 addImage("brown.png");
 addImage("gold.png");
 addImage("green.png");
 
 // compensate for difference in camera and projector aspect ratio
 // I am unsure about the math here...
 float aspectKinect = 640.0 / 480.0;
 float aspectNative = (float) displayWidth / displayHeight;
 aspectConst = aspectKinect / aspectNative;
}

void draw() {
 background(0);
 
 if (ks.isCalibrating()) {
 kinect.update();
 
 int scaledX = (int) (displayWidth * aspectConst * scaleX);
 int scaledY = (int) (displayHeight * scaleY);
 
 image(kinect.rgbImage(), offsetX, offsetY, scaledX, scaledY);
 
 fill(0, 0, 0);
 stroke(220);
 strokeWeight(3);
 rect(displayWidth - 250, 50, 210, 550, 10);
 
 int textLeft = displayWidth - 230;
 int textTop = 240;
 
 fill(255);
 textSize(12);
 text("Texture:", textLeft-5, 75);
 text("Controls:", textLeft-5, textTop);
 
 textTop += 24;
 text("w/q - offset x, up/down", textLeft, textTop);
 textTop += 20;
 text("s/a - offset y, up/down", textLeft, textTop);
 textTop += 20;
 text("r/e - scale x, up/down", textLeft, textTop); 
 textTop += 20;
 text("f/d - scale y, up/down", textLeft, textTop);
 textTop += 20;
 text("+/- or </> - change texture", textLeft, textTop);
 textTop += 20;
 text("n - new panel", textLeft, textTop);
 textTop += 20;
 text("l - load layout", textLeft, textTop);
 textTop += 20;
 text("o - save layout", textLeft, textTop);
 textTop += 20;
 text("k - load settings", textLeft, textTop);
 textTop += 20;
 text("i - save settings", textLeft, textTop);
 textTop += 20;
 text("[space] - toggle calibration", textLeft, textTop);
 textTop += 40;
 text("offset x: " + offsetX, textLeft, textTop);
 textTop += 20;
 text("offset y: " + offsetY, textLeft, textTop);
 textTop += 20;
 text("scale x: " + scaleX, textLeft, textTop);
 textTop += 20;
 text("scale y: " + scaleY, textLeft, textTop);
 
 image(images[currentImage], displayWidth - 200, 100, 100, 100);
 }
 
 for(int i = 0; i < surfaceIndex; i++) {
 surfaces[i].render(textures[i]);
 }
}

void addImage(String imageFile) {
 images[imageIndex++] = loadImage(imageFile);
}

void addTexture() {
 textures[surfaceIndex] = createGraphics(200, 200, P2D);
 textures[surfaceIndex].beginDraw();
 textures[surfaceIndex].image(images[currentImage], 0, 0, 100, 100);
 textures[surfaceIndex].endDraw();
}

void addSurface() {
 surfaces[surfaceIndex] = ks.createCornerPinSurface(100, 100, 20);
 surfaces[surfaceIndex].moveTo(mouseX, mouseY);
 addTexture();
 surfaceIndex++;
}

void doOffset(int x, int y) {
 offsetX += x;
 offsetY += y;
 
 for(int i = 0; i < surfaceIndex; i++) {
 surfaces[i].moveTo(surfaces[i].x + x, surfaces[i].y + y);
 }
}

void loadSettings() {
 float[] settingsArray = float(split(loadStrings("settings.txt")[0], ","));
 offsetX = int(settingsArray[0]);
 offsetY = int(settingsArray[1]);
 scaleX = settingsArray[2];
 scaleY = settingsArray[3];
}

void saveSettings() {
 String[] settingsStr = new String[1];
 settingsStr[0] = "" + offsetX + "," + offsetY + "," + scaleX + "," + scaleY;
 saveStrings("settings.txt", settingsStr);
}

void keyPressed() {

 switch(key) {
 case 'w':
 doOffset(5,0);
 break;
 
 case 'q':
 doOffset(-5,0);
 break;
 
 case 's':
 doOffset(0,5);
 break;
 
 case 'a':
 doOffset(0,-5);
 break;
 
 case 'r':
 scaleX += 0.01;
 break;
 
 case 'e':
 scaleX -= 0.01;
 break;

 case 'f':
 scaleY += 0.01;
 break;
 
 case 'd':
 scaleY -= 0.01;
 break;
 
 case 'n':
 addSurface();
 break;
 
 case '.':
 case '=':
 if ((currentImage + 1) == imageIndex) {
 currentImage = 0;
 } else {
 currentImage++;
 }
 break;
 
 case ',':
 case '-':
 if (currentImage == 0) {
 currentImage = imageIndex - 1;
 } else {
 currentImage--;
 }
 break;
 
 case ' ':
 // enter/leave calibration mode, where surfaces can be warped
 // & moved
 ks.toggleCalibration();
 break;

 case 'l':
 // TODO: Make this actually work!
 // loads the saved layout
 ks.load();
 break;

 case 'o':
 // saves the layout
 ks.save();
 break;
 
 case 'i':
 saveSettings();
 break;

 case 'k':
 loadSettings();
 break;
 }
} 

 
There is some code in here to link it to a Kinect to map the space, which I had hoped to generate the surfaces with, but alas that is a little over my head and skill level. If you get it to work let me know.
 

Test Mapping

Since I was building the project Play, which is an interactive physical/ digital collaborative fort, I wanted to see if given the resources if it was feasible to build. I also wanted to understand the benefits, caveats of the technology, and impact that projection mapping might have on the participants. The following are images and a video of the tests.
 

Projection mapping setupThis shows the projection with the alignment grid and the green corners.
ProjectionThe result outside of calibration mode.

 

The tests where setup with a home cinema projector, a laptop, and some handy ikea storage boxes.