Translate

Archives

HTML5 Canvas Element Rubber Banding

Some time ago I wanted to provide rubber banding functionality for a Firefox add-on which I was developing that used the HTML5 canvas element as a drawing surface. Not having previously implemented rubber banding on a HTML5 canvas element, but having done so in the X Window system and in Microsoft Windows, I thought that it would be something that would be relatively trivial to implement. Just find the appropriate routines, plug them into my application and and declare victory. Much to my surprise when I did an Internet search on the topic I found very little useful information on the subject.

As anybody who knows me well will attest, I like a challenge – the more difficult the better – and so just went ahead and developed my own version of rubber banding on the HTML5 canvas element. To simplify the usual develop/test/debug cycle (which is a more time consuming than usual when you are developing a Firefox add-on), I decided to write a small XULrunner application which would support rubber banding and demonstrate a number of operations on the area within the rubber banded box. I chose to implement image graying, embossing, flipping and inverting to see how expensive these operations would be to implement in JavaScript. There are many JavaScript implementations of these image manipulation operations available on the Internet but, again, none that I could easily find that support operations on parts of an image rather than the whole image.

Why XULrunner? Simply because it is tightly linked to Firefox and other Mozilla projects such as ThunderBird. If rubber banding worked within a XULrunner application, there was a very good chance that it would work within a Firefox add-on and other Mozilla applications with minor modifications. In fact, this proved to be the case.

You can try out the application here. Note it requires a Firefox browser. Other browsers do not understand a XULrunner application and simply regard the application as a file to be downloaded.

Original ImageModified Image

The image on the left is the unedited image and the image on the right is an example of an emboss operation, a flip operation, an invert operation and a gray operation which still has the rubber banding around the selected area.

Here is the full source code for the XULrunner application:

<?xml version="1.0"?>
<?xml-stylesheet href="chrome://global/skin/" type="text/css"?>
<?xml-stylesheet href="peaches.css" type="text/css"?>

<window id="peaches"
    title="Peaches"
    onload="Demo.init();" 
    xmlns:html="http://www.w3.org/1999/xhtml" 
    xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
 
    <script type="application/x-javascript">
    <![CDATA[ 
        var Demo = {
           canvas : null,
           ctx : null,       
 
           startX: null,
           startY: null,
           endX: null,
           endY: null,
           fImage: null,

           iData: null,
           iDataTop: null,
           iDataBottom: null,
           iDataLeft: null,
           iDataRight: null,
           iDataCoord: null,
   
           init : function() {
              this.canvas = document.getElementById("canvas");
              this.ctx = this.canvas.getContext("2d");
              this.startX = 0;
              this.startY = 0;
              this.endX = 0;
              this.endY = 0;
             
              this.ctx.strokeStyle = "red"; 
              this.ctx.font = "10pt Arial Bold";
              this.ctx.lineWidth = 1;

              this.fImage = document.getElementById("testimage");
              this.ctx.drawImage(this.fImage,0,0);
             
              this.canvas.addEventListener("mousedown", this.doDrawStart, false);
           },
 

           doDrawStart : function(event) {
               if (Demo.iDataTop) {
                    Demo.ctx.putImageData(Demo.iDataTop,    Demo.startX - 1, Demo.startY - 1,
                                                            Demo.startX - 1, Demo.startY - 1, 
                                                            Demo.iDataTop.width, Demo.iDataTop.height);
                                        
                    Demo.ctx.putImageData(Demo.iDataLeft,   Demo.startX - 1, Demo.startY - 1, 
                                                            Demo.startX - 1, Demo.startY - 1, 
                                                            Demo.iDataLeft.width, Demo.iDataLeft.height);

                    Demo.ctx.putImageData(Demo.iDataRight,  Demo.endX - 1, Demo.startY - 1, 
                                                            Demo.endX - 1, Demo.startY - 1, 
                                                            Demo.iDataRight.width, Demo.iDataRight.height);

                    Demo.ctx.putImageData(Demo.iDataBottom, Demo.startX - 1, Demo.endY - 1, 
                                                            Demo.startX - 1, Demo.endY - 1, 
                                                            Demo.iDataBottom.width, Demo.iDataBottom.height);

                    Demo.ctx.putImageData(Demo.iDataCoord, 20, 579,
                                                            Demo.iDataCoord.width, Demo.iDataCoord.height);
              }
        
              Demo.iData = Demo.ctx.getImageData(0, 0, Demo.canvas.width, Demo.canvas.height);
              
              Demo.startX = (event.clientX - event.target.parentNode.boxObject.x);
              Demo.startY = (event.clientY - event.target.parentNode.boxObject.y);
              Demo.endX = Demo.startX;
              Demo.endY = Demo.startY;
 
              Demo.canvas.addEventListener("mousemove", Demo.doDrawUpdate, true);
              Demo.canvas.addEventListener("mouseup", Demo.doDrawStop, true);
           },


           doDrawUpdate : function(event) {
              Demo.endX = (event.clientX - event.target.parentNode.boxObject.x);
              Demo.endY = (event.clientY - event.target.parentNode.boxObject.y);

              Demo.ctx.putImageData(Demo.iData, 0, 0);
              Demo.ctx.strokeRect(Demo.startX + 1, Demo.startY + 1, 
                 Demo.endX - Demo.startX - 1, Demo.endY - Demo.startY - 1 ); 
           },

 
           doDrawStop : function(event) {
               if (Demo.startX < 1) Demo.startX = 1;
               if (Demo.startY < 1) Demo.startY = 1;
               if (Demo.startX > Demo.canvas.width) Demo.startX = Demo.canvas.width - 1;
               if (Demo.startY > Demo.canvas.height) Demo.startY = Demo.canvas.height - 1;

               Demo.endX = (event.clientX - event.target.parentNode.boxObject.x);
               Demo.endY = (event.clientY - event.target.parentNode.boxObject.y);
               if (Demo.endX < 1) Demo.endX = 1;
               if (Demo.endY < 1) Demo.endY = 1;
               if (Demo.endX > Demo.canvas.width) Demo.endX = Demo.canvas.width - 1;
               if (Demo.endY > Demo.canvas.height) Demo.endY = Demo.canvas.height - 1;

               Demo.canvas.removeEventListener("mousemove", Demo.doDrawUpdate, true);
               Demo.canvas.removeEventListener("mouseup", Demo.doDrawStop, true);

               Demo.ctx.putImageData(Demo.iData, 0, 0);
              
               Demo.iDataTop    = Demo.ctx.getImageData(Demo.startX - 1, Demo.startY - 1, 
                                                        Demo.endX - Demo.startX + 2, 4);
               Demo.iDataLeft   = Demo.ctx.getImageData(Demo.startX - 1, Demo.startY - 1, 
                                                        4, Demo.endY - Demo.startY + 2);
               Demo.iDataRight  = Demo.ctx.getImageData(Demo.endX - 1, Demo.startY - 1, 
                                                        4, Demo.endY - Demo.startY + 2);
               Demo.iDataBottom = Demo.ctx.getImageData(Demo.startX - 1, Demo.endY - 1, 
                                                        Demo.endX - Demo.startX + 2, 4);

               if (Demo.startX == Demo.endX || Demo.startY == Demo.endY) { 
                   return;
               }

               Demo.ctx.strokeRect(Demo.startX + 1, Demo.startY + 1, 
                    Demo.endX - Demo.startX - 1, Demo.endY - Demo.startY - 1);

               var cString = '(' + Demo.startX + ',' + Demo.startY + ') (' + Demo.endX + ',' + Demo.endY + ')';
               var cWidth = Demo.ctx.measureText(cString).width;
               Demo.iDataCoord = Demo.ctx.getImageData(20, 579, cWidth, 14);

               Demo.ctx.fillStyle = "red";
               Demo.ctx.fillText(cString, 20, 590);                
           },

  
           doInvertImage : function() { 
              if (!this.startX) {
                  var imagedata = this.ctx.getImageData(0, 0, this.canvas.width, this.canvas.height);
              } else {
                  var imagedata = this.ctx.getImageData(this.startX + 1, this.startY + 1, 
                                  this.endX - this.startX - 1, this.endY - this.startY - 1);
              }
              var pix = imagedata.data;

              for (var i = 0, n = pix.length; i < n; i += 4) {
                 pix[i]   = 255 - pix[i];    // red
                 pix[i+1] = 255 - pix[i+1];  // green
                 pix[i+2] = 255 - pix[i+2];  // blue 
              }
              if (!this.startX) {
                  this.ctx.putImageData(imagedata, 0, 0);
              } else {
                  this.ctx.putImageData(imagedata, this.startX + 1, this.startY + 1);
                  this.ctx.strokeRect(this.startX + 1, this.startY + 1, 
                     this.endX - this.startX - 1, this.endY - this.startY - 1 ); 
              }
           },


           doGrayImage : function() {
              if (!this.startX) {
                  var imagedata = this.ctx.getImageData(0, 0, this.canvas.width, this.canvas.height);
              } else {
                  var imagedata = this.ctx.getImageData(this.startX + 1, this.startY + 1, 
                                  this.endX - this.startX - 1, this.endY - this.startY - 1);
              }
              var pix = imagedata.data;

              for (var i = 0, n = pix.length; i < n; i += 4) {
                 var lum = Math.round(pix[i] * 0.3 + pix[i+1] * 0.59 + pix[i+2] * 0.11);
                 pix[i]   = lum;     // red
                 pix[i+1] = lum;     // green
                 pix[i+2] = lum;     // blue 
              }
              if (!this.startX) {
                  this.ctx.putImageData(imagedata, 0, 0);
              } else {
                  this.ctx.putImageData(imagedata, this.startX + 1, this.startY + 1);
                  this.ctx.strokeRect(this.startX + 1, this.startY + 1, 
                     this.endX - this.startX - 1, this.endY - this.startY - 1 ); 
              }
           },

  
           doFlipImageHorz : function() {
              if (!this.startX) {
                  var imagedata = this.ctx.getImageData(0, 0, this.canvas.width, this.canvas.height);
                  var height = this.canvas.height;
                  var width = this.canvas.width;
              } else {
                  var imagedata = this.ctx.getImageData(this.startX + 1, this.startY + 1, 
                                  this.endX - this.startX - 1, this.endY - this.startY - 1);
                  var height = this.endY - this.startY - 1;
                  var width = this.endX - this.startX - 1;
              }
              var pix = imagedata.data;

              for (var i = 0; i < height; i++) {
                  for (var j = 0; j < width/2; j++) {
                     var index = (i*4) * width + (j*4);
                     var mIndex = ((i+1)*4) * width - ((j+1)*4);
                     for (var p = 0; p < 4; p++) {
                        var temp = pix[index + p];
                        pix[index + p] = pix[mIndex + p];
                        pix[mIndex + p] = temp;
                     }
                 }
              }

              if (!this.startX) {
                  this.ctx.putImageData(imagedata, 0, 0);
              } else {
                  this.ctx.putImageData(imagedata, this.startX + 1, this.startY + 1);
                  this.ctx.strokeRect(this.startX + 1, this.startY + 1, 
                     this.endX - this.startX - 1, this.endY - this.startY - 1); 
              }
           },


           doEmboss : function() {
              if (!this.startX) {
                  var imagedata = this.ctx.getImageData(0, 0, this.canvas.width, this.canvas.height);
                  var height = this.canvas.height;
                  var width = this.canvas.width;
              } else {
                  var imagedata = this.ctx.getImageData(this.startX + 1, this.startY + 1, 
                                  this.endX - this.startX - 1, this.endY - this.startY - 1);
                  var height = this.endY - this.startY - 1;
                  var width = this.endX - this.startX - 1;
              }
              var pix = imagedata.data;

              var y = height;
              do {
                  var offsetY = (y - 1) * width * 4;
                  var offsetYPrev = (y - 2) * width * 4;
                  if (y == 1) offsetYPrev = offsetY;

                  var x = width;
                  do {
                      if (x < width && y < height) {
                          var offset = offsetY + (x - 1) * 4;
                          var offsetPrev = offsetYPrev + (x - 1) * 4;
                          if (x == 1) offsetPrev = offset;

                          var deltaR = pix[offset]     - pix[offsetPrev - 4];
                          var deltaG = pix[offset + 1] - pix[offsetPrev - 3];
                          var deltaB = pix[offset + 2] - pix[offsetPrev - 2];

                          var delta = deltaR;
                          if (Math.abs(deltaG) > Math.abs(delta)) delta = deltaG;
                          if (Math.abs(deltaB) > Math.abs(delta)) delta = deltaB;

                          var grey = 180 - delta;
                          if (grey < 0) grey = 0;
                          if (grey > 255) grey = 255;

                          pix[offset]     = grey;
                          pix[offset + 1] = grey;
                          pix[offset + 2] = grey;					                      }
                  } while (--x);
		  } while (--y);
      
              if (!this.startX) {
                  this.ctx.putImageData(imagedata, 0, 0);
              } else {
                  this.ctx.putImageData(imagedata, this.startX + 1, this.startY + 1);
                  this.ctx.strokeRect(this.startX + 1, this.startY + 1, 
                     this.endX - this.startX - 1, this.endY - this.startY - 1); 
              }
           },

           doLoadImage : function() {
              this.ctx.drawImage(this.fImage, 0, 0); 
              this.startX = 0;
              this.startY = 0;
              this.endX = 0;
              this.endY = 0;
              this.iDataTop = null;
           },

       };
   ]]> 
   </script>

    <vbox align="center" flex="1" style="overflow:auto" >
       <groupbox style="width:460px; height:680px; border:1px solid black;">
           <caption label="Peaches"/>
           <vbox align="center" flex="1">
           <hbox>
               <button label="Load" oncommand="Demo.doLoadImage();" tooltiptext="Load image of dog" />
               <button label="Invert" oncommand="Demo.doInvertImage();" tooltiptext="Image all or rubberbanded area" />
               <button label="Gray" oncommand="Demo.doGrayImage();" tooltiptext="Gray (grey) all or rubberbanded area" />
               <button label="Flip" oncommand="Demo.doFlipImageHorz();" tooltiptext="Flip all or rubberbanded area" />
               <button label="Emboss" oncommand="Demo.doEmboss();" tooltiptext="Emboss all or rubberbanded area" />
           </hbox>
           <hbox>
               <html:img id="testimage" src="./peaches.png" style="display:none;"></html:img>
           </hbox>
           <hbox>
               <html:canvas id="canvas" width="400" height="600" style="background-color:#fff; border:3px solid black;"></html:canvas>
           </hbox>
           </vbox>
        </groupbox>
     </vbox>

</window>


All the rubber banding functionality is contained within the three routines doDrawstart, do DrawUpdate and doDrawStop. The code should be understandable by any experienced JavaScript programmer. I did not use the classical XOR method of generating a visible rubber band. Instead I chose to use strokeRect to output a solid red rubber band. To remove the rubber band, I restored the pixels under the rubber band (which I saved prior to drawing the rubber band).

This application is not bulletproof. It is just a proof of concept. It has some minor bugs in it that I am aware of. For example, the emboss routine does not handle the first left pixel and the top row of pixels correctly. You can see this when you select the emboss option and have not yet removed the red rubber band.

Looking behind the scenes (See nsIDOMCanvasRenderingContext2D.idl) at how support for the canvas element is implemented in Firefox , I discovered that canvas is implemented on top of the popular Cairo 2D vector graphics library. The Cairo graphics library was originally developed for the X Window System by Keith Packard and Carl Worth but has since then been expanded to a more general purpose graphics library.

Finally, for those readers who are unfamiliar with the HTML5 canvas element, Mozilla has a good introductory tutorial. Opera also has good tutorial on the subject.

Enjoy!
 

Comments are closed.