Test Python in VSTS
Microsoft’s Team Data Science Process Documentation has a lot of info/opinions on how to do data science with Azure.
That also includes quite a general setup for testing Python code using custom Docker containers and a Kubernetes orchestrator.
However, for Python there is also a more “as a Service”-like solution that I will cover here.
Making a Python package
Python is not a core language for me, so I have relied on the official docs for making modules and packages as well as the popular post Packaging a python library.
I have made a small Python package that runs on both Python 2.7 and 3.6.
The package is called
vsts and is in a folder called
The package is very simple, containing only one module for doing simple arithmetic:
│ └── test_arithmetic.py
The arithmetic module is also very simple, containing only two functions:
def add(x, y):
Return the sum of two numbers
return(x + y)
def subtract(x, y):
Return the difference of two numbers
return(x - y)
__init__.py file imports these two functions and make them available directly in the
(For any other Python noobs: The dot in front of
from .arithmetic import add, subtract
arithmetic is necessary in Python 3, but not in Python 2.)
Finally, there are two simple tests in
assert vsts.add(1, 1) == 2
assert vsts.subtract(1, 1) == 0
The tests are for the pytest framework.
I use pytest because it can write the test results in formats that VSTS can import.
Code coverage is computed with the package pytest-cov.
setup.py script contains information for making the actual package (https://docs.python.org/3/distutils/setupscript.html):
# -*- encoding: utf-8 -*-
from setuptools import setup
description='Demonstrating automated testing on VSTS',
author='Robert Dahl Jacobsen',
run_tests.ps1 is covered in the VSTS section
To run the tests locally, navigate to the root directory of the package and execute the following (in PowerShell on Windows or any shell on *nix):
This command reads as follows: Run the test scripts in the folder
python -m pytest tests --cov=vsts
tests and compute code coverage for the folder
The results are as follows:
$ python -m pytest tests --cov=vsts
======================== test session starts ========================
platform linux2 -- Python 2.7.12, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
rootdir: /home/robert/Documents/python/vsts, inifile:
collected 2 items
tests/test_arithmetic.py .. [100%]
---------- coverage: platform linux2, python 2.7.12-final-0 ----------
Name Stmts Miss Cover
vsts/__init__.py 1 0 100%
vsts/arithmetic.py 4 0 100%
TOTAL 5 0 100%
===================== 2 passed in 0.03 seconds ======================
All tests run and all non-commented lines lines in
arithmetic.py are executed during the tests.
Test in VSTS
In VSTS the project repository is called
Testing in VSTS is found under the
Build & Release tab.
Create a new
Build Definition under
The bare minimum for a compiled language is a build, but we can also add more steps.
Here are the steps I use:
Continuous integration is enabled under the
Phase 1 I use a “Hosted VS2017” under “Agent queue”.
This executes the code on a computer that I don’t have to maintain and thus eliminates the need for creating and running the Docker containers mentioned in the beginning.
Note that only selected languages are available on the hosted solutions.
PowerShell Script step is a custom build.
PowerShell scripts can be executed in two ways:
As an inline script where you write the script directly in the build definition. This is an easy solution for small scripts, but the script itself is not version controlled.
The other option is to include the script in the repository, which is what I use here in
python -m pip install .
python -m pip install -U pytest pytest-cov
python -m pytest tests --junit-xml=test-results.xml --cov=vsts --cov-report=xml:coverage.xml
By default the path is that of the script, but this can be changed in the “Advanced” section or programmatically through the build variable
First we test if the package actually installs correctly.
The packages listed in
setup.py are not installed with the first pip command and are therefore installed manually.
The last line that run the tests reads as follows:
Run the test scripts in the
tests folder and write the results in the JUnit XML format in the file
test-results.xml; compute code coverage for the folder
vsts and write the results as
When choosing “xml” for the
cov-report it is in the Cobertura format.
See all options with
Sometimes warnings (such as notification about an outdated
pip) are written to the PowerShell error stream:
So even though your test scripts runs, no results are collected in the subsequent tasks.
To circumvent this the box “Fail on Standard Error” must be unchecked in the Advanced section of PowerShell Script.
After running the PowerShell script for testing, we import test results and import code coverage into VSTS.
For the test results I use the default values:
- Test result format:
- Test result files:
For the code coverage I use these values:
- Code coverage tool:
- Summary file:
After running a test we get an output that looks like the following: