Using Selenium for getting a full image of a page.

The last few weeks I've been working on and off on a patch to acquire complete images of websites using Selenium-rc, now it is done.

The patch builds on the work done by Robert Zimmermann, but takes it a few steps further, creating two new selenium commands:

1. initScreenshots creates a calibration image that is used to define the area that will be saved into the image.


2. You can then navigate to the spot where you want to take a picture, and then call getFullScreenshot to get a full image.

Both commands maximize the window and getFullScreenshot also takes multiple screenshots of the page (while scrolling it) and then merges these together.

You can see a sample image taken of http://www.nytimes.com/ here (firefox) . I'm having a smal problem getting IE to do proper scrolling, but that should be fixable.

Compiling and running it yourself

I've attached the patches to bug #395. As a further help, you can find ready compiled jars for testing here: http://folk.uio.no/tarjeih/selenium/. You need to compile your tests using the selenium-java-client-driver-0.9.3-SNAPSHOT.jar file.

To run the server, you need to increase the amount of heap space to be able to merge the images. Here's the commandline I use to run this on windows:

java -Xmx256m -jar "Z:\selenium-server-0.9.3-SNAPSHOT-standalone.jar" -multiWindow


Further work

The basic patch is fairly clean, although some of the merge tests depend on quite a few images and also contains some Unixims, mainly that it saves files to /tmp.

Sample tests:


import java.awt.image.BufferedImage;

import java.io.File;

import javax.imageio.ImageIO;

import junit.framework.TestCase;

import com.thoughtworks.selenium.DefaultSelenium;

import com.thoughtworks.selenium.Selenium;



public class TestRemoteFullImage extends TestCase {

  private Selenium selenium;

  public void testRemoteImage() throws Exception {

  String url = "http://www.nytimes.com/";  

  selenium = new DefaultSelenium("wintest", 4444, "*firefox", url);

        selenium.start();

        System.out.println(selenium.initScreenshots());

        selenium.open(url);

        selenium.waitForPageToLoad("20000");

        BufferedImage rendImage = selenium.getFullScreenshot("png");

        File file = new File("/tmp/imresult.png");

        assertNotNull(rendImage);

        ImageIO.write(rendImage, "png", file);

  }

  public void testRemoteImageOnIE() throws Exception {

    String url = "http://www.nytimes.com/";

    selenium = new DefaultSelenium("wintest", 4444, "*iexplore", url);

        selenium.start();

        System.out.println(selenium.initScreenshots());



        selenium.open(url);

        selenium.waitForPageToLoad("20000");

        BufferedImage rendImage = selenium.getFullScreenshot("png");

        File file = new File("/tmp/iexporeresult.png");

        assertNotNull(rendImage);

        ImageIO.write(rendImage, "png", file);



  }

}








Java2html

Selenuim now has captureImage()

I just noticed over the holidays that Selenium has got support for retrieving images of a browser. It is still a patch though, but I've given it a spin and it works quite well. I'm now trying to extend it to capture the whole "page" not the part shown by the browsers viewport. More on that in a later post. Here's the basic code I use to get a picture:

import java.awt.image.BufferedImage;
import java.awt.image.RenderedImage;
import java.io.File;
import java.io.IOException;
import javax.imageio.ImageIO;
import com.thoughtworks.selenium.DefaultSelenium;
import com.thoughtworks.selenium.Selenium;
import junit.framework.TestCase;

public class SimpleScreenshotTest extends TestCase {
   private Selenium selenium;

   public void setUp() throws Exception {
       String url = "http://www.google.com";
       // browsers:
       //selenium = new DefaultSelenium("wintest", 4444, "*iexplore", url);
       selenium = new DefaultSelenium("wintest", 4444, "*firefox", url);
       selenium.start();
   }

   protected void tearDown() throws Exception {
       selenium.stop();
   }

   /**
    * The normal google test, just with a screenshot twist.
    */

   public void testGoogle() throws Throwable {
       selenium.open("http://www.google.com/webhp?hl=en");

       assertEquals("Google", selenium.getTitle());

       /* use maximize window to be able to get as much of the viewport as possible. */
       selenium.windowMaximize();
       selenium.waitForPageToLoad("5000");
       // note: getScreenshot does not validate the inputparameters.
       RenderedImage rendImage = selenium.getScreenshot("png");
       if (rendImage == null) {
           fail("Rendered image null!");
       }
       File file = new File("/tmp/newimage-.png");
       ImageIO.write(rendImage, "png", file);
   }

The next thing now will be to look into creating shots of the whole page.

If someone needs help building Selenium with the patch, post to the comments and I'll do a quick writeup of how to compile selenium-rc.

Grabbing the complete picture

One of the goals of my project is to make the process of making HTML+CSS code from a designers sketch picture test driven.

To do this, I need to strip out the margins and headers of the various images I get from browsershots.org (my preferred way right now of getting pictures of websites).

This isn't easy, but I am working on different approaches starting of by trying to segment on the color of the surrounding element. My goal is to use as little information as possible about the image to extract what I want.

A problem with the current images is that each image includes the whole browser window including scrollbars, addressbars and other browser specific stuff. It is a goal to be able to remove as much of this clutter as possible. We want a clean image of just the rendered page.

As a test, I started with four images from four different browsers, firefox 1.5, firefox 2.0, safari and opera. Somehow I forgot to include IE in the dataset.

You can find the images here:
http://folk.uio.no/tarjeih/matlab/

For now I've done removal of horizontal components in the image. To do this, I used matlab to get the variance of each horizontal line in the image. Then I set a cutoff point wrt to variance and also cutoff points that I was only interested in lines 120 or 50 pixels from the top or bottom respectively.

For each picture I got two sets of lines, one for the lines on top with variance bellow the set cutoff point and one for the lines close to the bottom.

I then used the line closest to the center for each of these sets as the line where I thought the webpage started.

The images in the directory show the results.

The file .png is the original image and the file cut-.png shows the image with red lines drawn where I found lines with variance bellow the cutoff point.

Files named figure-.png.eps shows the two images side by
side.

Relevant matlab code is also in the directory:
- cutPic.m is the function that finds the cut off points.
- getCutPicLines.m is the script that goes through the images and
crops them.
- colorLines is a script to color the relevant lines in the image.
- tmp_rmLines .m is a starting attempt at making the script work in
both dimensions.

Questions/Problems/Ideas/Comments:
* It is fairly obvious that this approach works best at the top and bottom.
* I'm wondering if I should use information about which browser this is when cropping the image. I was hoping I didn't have to, but I seem to be wrong.
* I will need an approach to deciding if there is stuff surrounding the image, esp if there is a scrollbar in the image.

Browserbeaten - what's in a name?

Why browserbeaten? Because nitwit, bit, imtest and a host of other names on blogspot were taken.

Browserbeaten also refers to the state of many web developers today. We make cool and jazzy AJAX based sites with lots of nice CSS and then the whole shebang collapses as the boss says "Why doesn't this work with Safari?" and you have to work double overtime to fix browserbugs just before release . The browser beat you.

In my view, browsers is the new portability challenge for developers. Therefore we need to find ways to test our products across a lot of browsers fast. I'm a great fan (if not always a perfect practitioner) of Test driven development (TDD).

Right now I am working on a masters thesis that tries to combine techniques from computer vision with ideas from test driven development to see if it is possible to make a tool that tests the design of a web page in different browsers.

This blog is my dumping ground for ideas, links and that I will use to develop my masters thesis.