Visual Testing - Scriptworks Image Comparison Features

This article looks at the Image Comparison features for performing Visual Testing in Scriptworks.

Introduction

Visual Testing is the process of using Image Comparison techniques during test execution as a mechanism for validating the way an Application Under Test (AUT) looks compared to how it looked on a previous execution.

Visual Testing is often used as an additional check mechanism in test workflows which contain other checks for values and content, often picking up bugs that these do not.

Process

The process is as follows:

1. Creation of Baseline Images for an AUT Screen or Element at given points during the test workflow (this is done automatically for new Visual Checks for a project during initial test execution).
2. Manual confirmation that the Baseline Images are correct.
3. Capturing Actual Images during subsequent execution and Comparing with the Baseline Images automatically.

If visual differences are found (above a given threshold) during Step 3 the following process is followed:

1. Create a Diff Image - highlighting where the visual differences were detected.
2. Store the Actual Image as a Candidate for a new baseline.
3. Report that visual differences were found.

This is highlighted in the diagram below:

 

Adding a Visual Check to a Test

The Visual Testing Blocks can be found in the Test - Visual Testing (BETA) category as shown below:

A Visual Check can be added at any point during a test workflow by dragging the check screen block into the test at the appropriate point.

It is best practice to add a Synchronisation step prior to this e.g. A Wait for Browser Activity block to ensure the AUT has completed any rendering activity.

The example below shows the check screen block. The Check name has been given as "DefaultState"

This will perform a visual check on the visible AUT screen at the point in the workflow - The Baseline Image will be stored as a file named DefaultState.png e.g. for the test above for:

http://decohere.herokuapp.com/planets

This will result in the following being captured:

In subsequent test runs, Actual Images will be compared with this Baseline.

 

Adding a Visual Test for an Element

Visual Tests can be performed on Individual Elements within an Application by using the check element block along with the Selector for an Element from the Elements list.

First drag in the check element block:

Then the Element Selector block, and fill in the check name:

This will create a Visual Check on the Calculate button only and the Baseline Image will be stored as a file named CalculateButton.png.

This will result in the following being captured:

In subsequent test runs, Actual Images will be compared with this Baseline.

 

Blocking areas from a Visual Test

It is desirable in some circumstances to ignore or block certain areas of an AUT screen from an Image Comparison during Visual Testing. This may be in the circumstances where:

• Time or Date values are shown that may change in subsequent runs
• Dynamically calculated or created values are shown e.g. Account Number
• Dynamic content is shown e.g. Rolling Adverts

This can be achieved in Scriptworks Visual Testing using the block element block.

First drag in the block element block:

Then drag in the Selector for the Element you wish to ignore e.g. in this example the Planet Logo:

The bounding are of the Element is ignored from subsequent Image Comparisons as shown below where the logo is blocked out:

More than one area can be blocked by adding additional block element blocks to the test.

 

Reporting

Visual checks are reported in the standard Scriptworks results.

Passed Checks

A passed check will show an image of the actual AUT screen or element:

The Image can be clicked on to enlarge it.

 

 Failed Checks

Failed checks will show 3 images, the baseline image, the diff image and the actual image:

The Diff Image highlights the areas that are different in Pink, and the number of pixels difference is displayed. Again the image can be enlarged by clicking:

Here we can see the fields now contain values and the Calculate button has changed appearance. The default state is not as expected.

Saving a New Baseline

When a check fails a Baseline Candidate Image is stored in the Baseline folder. This will be stored with the filename <check name>_Candidate.png - The file can be assessed manually and if acceptable the old baseline can be replaced by this image by deleting the old <check name>.png file and renaming the <check name>_Candidate.png file to <check name>.png.

Baseline Image Storage

Baseline Images are stored on the local filesystem where execution takes place. This can be any file location e.g. A network share or Github repository.

The location of the Baseline Image store defaults to the following:

<Username>/Documents/Scriptworks/VisualTesting/<ProjectName>/Baseline

Baseline Images are stored using the Visual Check Name given when adding a Visual Check to the script e.g.

The default location can be changed in the Visual Test Settings, described below:

 

Visual Test Settings (Optional)

The following can be configured for Visual Testing:

• Baseline Image Folder
• Threshold Percentage

An example of each is shown in the Script Parameters below:

Note the settings are case sensitive and are provided below:

Name:

visualTest

Type:

Object

Value:

{ "threshold": <value>, "baselineFolder": "<full path>" }

Threshold is a value between 0 and 1 to represent percentage of pixel differences e.g. 0.3 = 30% - Default is 0.

BaselineFolder is the full path to the folder e.g.

{ "threshold": 0.3, "baselineFolder": "//Users/tester/Documents/Scriptworks/VisualTesting/MyFolder/Baseline" }

The default location is:

<Username>/Documents/Scriptworks/VisualTesting/<ProjectName>/Baseline

 

Considerations for Browser Based Visual Testing

Web Apps are presented in Browsers which can have differing sizes and resolutions dependent on the OS or Device they are accessed from. Visual Testing requires that there is consistency in images in order to perform an accurate comparison. There are some recommendations below:

 

Set the Browser Size as part of the test

Use the set browser size block before performing a check screen Visual Test to ensure the browser is at a consistent size each time the test has run e.g.

This can be used for checking the state of the website at different browser resolutions in a single test e.g.

This will result in 3 different baseline images captured at different browser sizes.

 

Working with Retina or HDPI displays

Some displays including Apple Retina displays will scale the browser content to cater for these higher resolutions. If you are working with a HDPI or Apple Retina display images captured will be . Most browsers can be set to work at a consistent scale factor of 1 through the Run settings, desired capabilities. e.g. for Chrome: