pytest-needle

pytest-needle is a pytest implementation of needle.

It’s fairly similar to needle and shares much of the same functionality, except it uses pytest-selenium for handling the webdriver and implements needle as a fixture instead of having test cases inherit from needle’s base test class.

Installation

Install through pip:

pip install pytest-needle

Install from source:

cd /path/to/source/pytest-needle
python setup.py install

pytest-needle

Example

Example needle pytest implementation:

"""test_example.py
"""

from selenium.webdriver.common.by import By
import pytest

@pytest.mark.element
def test_example_element(needle):
    """Example for comparing individual elements

    :param NeedleDriver needle: NeedleDriver instance
    :return:
    """

    # Navigate to web page
    needle.driver.get('https://www.google.com')

    # Take an element screen diff
    needle.assert_screenshot('search_field', (By.ID, 'tsf'))

To create a baseline for all subsequent test run:

pytest --driver Chrome --needle-save-baseline test_example.py

After we have a baseline, to run test use:

pytest --driver Chrome test_example.py

Selecting a WebDriver

To control which browser to use, use --driver <BROWSER> from pytest-selenium. For example to change to browser to Firefox:

pytest --driver Firefox test_example.py

Setting the viewport’s size

You may set the size of the browser’s viewport using the set_viewport_size() on the needle fixture

def test_example_viewport(needle):

    # Navigate to web page
    needle.set_viewport_size(width=1024, height=768)

    # Rest of the test ...

You may also set the default viewport size for all your tests by using the command line argument --needle-viewport-size:

pytest --driver Chrome --needle-viewport-size "1024 x 768" test_example.py

Excluding areas

Sometimes areas on a web page may contain dynamic content and cause false negatives, or worse convince testers to raise the threshold at which changes are acceptable. You can instead choose to mask these areas to avoid the issue of consistently failing tests:

"""test_example.py
"""

from selenium.webdriver.common.by import By
import pytest


@pytest.mark.mask
def test_example_page_with_mask(needle):
    """Example for comparing page with a mask

    :param NeedleDriver needle: NeedleDriver instance
    :return:
    """

    # Navigate to web page
    needle.driver.get('https://www.google.com')

    # Take a entire page screen diff, ignore the doodle banner
    needle.assert_screenshot('search_page', threshold=60, exclude=[(By.ID, 'hplogo'), (By.ID, 'prm')])

In the case with Google’s home page the doodle banner frequently changes, so to visually regress day-to-day requires generating new baselines every time the banner is updated. Masking allows only the banner to be ignored while the rest of the page can be evaluated.

Advanced Settings

Engines

By default Needle uses the PIL engine (needle.engines.pil_engine.Engine) to take screenshots. Instead of PIL, you may also use PerceptualDiff or ImageMagick.

Example with PerceptualDiff:

pytest --driver Chrome --needle-engine perceptualdiff test_example.py

Example with ImageMagick:

pytest --driver Chrome --needle-engine imagemagick test_example.py

Besides being much faster than PIL, PerceptualDiff and ImageMagick also generate a diff PNG file when a test fails, highlighting the differences between the baseline image and the new screenshot.

Note that to use the PerceptualDiff engine you will first need to download the perceptualdiff binary and place it in your PATH.

To use the ImageMagick engine you will need to install a package on your machine (e.g. sudo apt-get install imagemagick on Ubuntu or brew install imagemagick on OSX).

File cleanup

Each time you run tests, Needle will create new screenshot images on disk, for comparison with the baseline screenshots. It’s then up to you whether you want to delete them or archive them. To remove screenshots from successful test use:

pytest --driver Chrome --needle-cleanup-on-success test_example.py

Any unsuccessful tests will remain on the file system.

File output

To specify a path for baseline image path use:

pytest --driver Chrome --needle-baseline-dir /path/to/baseline/images

Default path is ./screenshots/baseline

To specify a path for output image path use:

pytest --driver Chrome --needle-output-dir /path/to/output/images

Default path is ./screenshots

Generating HTML reports

To generate html reports use:

pytest --driver Chrome --html=report.html --self-contained-html

API Reference

Driver

pytest_needle.driver

class pytest_needle.driver.NeedleDriver(driver, **kwargs)

Bases: object

NeedleDriver instance

ENGINES = {'imagemagick': 'needle.engines.imagemagick_engine.Engine', 'pil': 'needle.engines.pil_engine.Engine', 'perceptualdiff': 'needle.engines.perceptualdiff_engine.Engine'}

assert_screenshot(file_path, element_or_selector=None, threshold=0, exclude=None)

Fail if new fresh image is too dissimilar from the baseline image

Note

From needle https://github.com/python-needle/needle/blob/master/needle/cases.py#L161

Parameters:

• file_path (str) – File name for baseline image
• element_or_selector – WebElement or tuple containing selector ex. (‘id’, ‘mainPage’)
• threshold – Distance threshold
• exclude (list) – Elements or element selectors for areas to exclude

Returns:

baseline_dir

Return baseline image path

Returns: Return type:str

cleanup_on_success

Returns True, if cleanup on success flag is set

Returns: Return type:bool

engine

Return image processing engine

Returns:

engine_class

Return image processing engine name

Returns: Return type:str

get_screenshot(element=None)

Returns screenshot image

Parameters:element (WebElement) – Crop image to element (Optional) Returns:

get_screenshot_as_image(element=None, exclude=None)

Parameters:

• element (WebElement) – Crop image to element (Optional)
• exclude (list) – Elements to exclude

Returns:

output_dir

Return output image path

Returns: Return type:str

save_baseline

Returns True, if save baseline flag is set

Returns: Return type:bool

set_viewport()

Set viewport width, height based off viewport size

Returns:

viewport_size

Return setting for browser window size

Returns: Return type:str

Exceptions

exceptions

exception pytest_needle.exceptions.ImageMismatchException(message, baseline_image, output_image, *args)

Bases: pytest_needle.exceptions.NeedleException

Image mismatch exception

exception pytest_needle.exceptions.NeedleException

Bases: exceptions.AssertionError

Base exception for pytest-needle

Plugin

pytest_needle.plugin

pytest_needle.plugin.get_image_as_base64(filename)

Open image from file as base64 encoded string

Parameters:filename (str) – File path Returns:

pytest_needle.plugin.is_failure(report)

True, if test failed

Parameters:report – Returns:

pytest_needle.plugin.needle(request, selenium)

Visual regression testing fixture

Parameters:

• request – pytest request
• selenium – Selenium web driver

Returns:

pytest_needle.plugin.pytest_addoption(parser)

Parameters:parser – Returns:

pytest_needle.plugin.pytest_runtest_makereport(item, call)

Add image diff to report

Parameters:

• item –
• call –

Returns:

Development

Installation

To install for development, simply run the following commands:

git clone https://github.com/jlane9/pytest-needle.git
cd pyest-needle
pip install -r requirements.txt
pip install -e .

Generating documentation

You can either use makefile:

cd docs
make html

Or you can use autobuild:

cd docs
sphinx-autobuild . _build/html/

Running Tests

To run tests you must first provide a base line to go against:

pytest --driver Chrome --needle-save-baseline test/

Then all runs afterwards can be just:

pytest --driver Chrome --pep8 pytest_needle --cov pytest_needle --cov-report term-missing test/

Miscellaneous

Special Thanks

Special thanks to BrowserStack for providing automated browser testing, at no charge, for this project and other open source projects like this. With over 1000+ device, browser and os versions combinations to choose from and integrations with Travis CI this project could not be successful without the hard work of the BrowserStack team and their continued support of the open source community.

Indices and tables

• Index
• Module Index
• Search Page