Tox is a generic virtualenv management and test command line tool that can be used to run tests in multiple environments (different versions of Python, different versions of the dependencies…). In this article, we will see the problems that arise when packages are managed using virtualenv and pip, and show how GNU Guix might be better suited for this task.
When tox + virtualenv fall short
Tox, along with virtualenv, has been a huge improvement in the development workflow of Python developers. Before tox, working with multiple versions of Python, or multiple versions of libraries, was so complex that most developers only used a single specific setup to run their tests. By providing control over complete Python development environments, tox has made it easy to replicate those environments and make Python development more tractable. Using tox, one may declare multiple environments using the tox.ini file and have tox handle their creation. For instance, in python-keystoneclient, five different setups might be used:
$ tox -l py26 py27 py34 pep8 bandit
By typing “tox -eENV”, one may “activate” the corresponding virtual environment (creating it if it is needed, and using it) and run the commands defined for this environment. For instance, “tox -epy26”, “tox -epy27” and “tox -epy34” will run the tests suite with Python 2.6, 2.7 and 3.4, while “tox -epep8” will instead run some coding style checks and “tox -ebandit” will look for security issues in the code base.
From a user perspective, tox is really convenient. If we dive into its internals, however, we can find implementation details that we are not pleased with. The main issue is that, by using virtualenv, tox cannot properly control what is happening inside the virtual environments. Let us go through a few real-life issues.
Pip cannot install packages that are not part of PyPI
Let us try to install the cffi package inside a virtual environment:
$ virtualenv libffi-test Running virtualenv with interpreter /usr/bin/python2 New python executable in libffi-test/bin/python2 Also creating executable in libffi-test/bin/python Installing setuptools, pip...done. $ source libffi-test/bin/activate (libffi-test)$ pip install cffi x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 \ -Wall -Wstrict-prototypes -fPIC -DUSE__THREAD -I/usr/include/ffi \ -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c \ -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o c/_cffi_backend.c:13:17: fatal error: ffi.h: No such file or directory #include^ compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Installing this package requires a fair amount of compilation. This means that users need a compiler, a C library, and various libraries that may not be installed on their systems (for instance, libffi-dev on Debian), which is an issue for two reasons:
- you sometimes have to do more than just running tox in order to create a working environment;
- different operating systems will provide different versions of the dependencies that cannot be found on PyPI: as a result not all developers may end up witnessing the same results when running the tests.
Tests are almost reproducible, but not quite
Tests should be reproducible. Running them inside virtual environments that are a bit isolated from the rest of the system is a good idea. Unfortunately, the outcome of the tests can still be affected by environment variables, external commands, external libraries… Let’s give some examples.
User-defined environment variables
Environment variables can change the outcome of a program:
$ LANG=C python3 -c "print('é')" Unable to decode the command from the command line: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 7: surrogates not allowed $ LANG=en_US.UTF-8 python3 -c "print('é')" é
Tox allows developers to specify *some* environment variables in tox.ini by using the setenv directive, but it cannot possibly have control over *all* of them when the tests are running; therefore, developers are likely to witness different outcomes when running the test suite.
External commands
Tox tries to warn you when a command it is not aware of is used:
[tox] envlist=py34 [testenv] commands= echo "Starting tests" python test.py $ tox -epy34 ... WARNING:test command found but not installed in testenv cmd: /bin/echo env: /tmp/fake-project/.tox/py34 Maybe forgot to specify a dependency? Starting tests py34 runtests: commands[1] | python test.py ...
Here, “/bin/echo” is not part of the virtualenv. The usual solution is to tell tox that we are aware of this, by using “whitelist_externals”:
$ cat tox.ini [tox] envlist=py34 [testenv] whitelist_externals=/bin/echo commands= echo "Starting tests" python test.py
Tox will silently ignore all “external” commands whitelisted in tox.ini. But now, the outcome of the tests depends on the version of “echo” installed on the machine where “tox” is called, and reproducibility cannot be guaranteed.
Some “external” commands might be used by the actual code, and tox cannot do anything about that. Here, the outcome of the tests depends on whether “git” is installed:
import os import sys def main(): return 0 if os.path.exists("/usr/bin/git") else 1 if __name__ == '__main__': sys.exit(main())
External libraries
As seen previously, users sometimes have to manually install dependencies because they are not available on PyPI. This happens with compilers, development packages (*-dev on Debian) or packages not written in Python. The Python Packaging Authority is aware of this issue and PEP497 (which is onlya draft at the time this article is being written) intends to specify how external dependencies should be declared in setup.py. Not all developers may end up with the same version of those packages, which adds a source of unreliability when running the tests.
Every language has a different package manager
Every language now comes with its own package manager: pip for Python, npm for Javascript, cpan for Perl… What if a single package manager could handle all packages and create isolated environments where they can be installed ?
Introducing Guix
A hackable package manager
GNU Guix is a functional package manager based on Nix. It supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. It is Free Software, developed by the GNU project and provides a Guile (an interpreter and compiler for the Scheme programming language) API to define and manipulate packages.
To quote the Guix manual:
The term functional refers to a specific package management discipline. In Guix,
the package build and installation process is seen as a function, in the
mathematical sense. That function takes inputs, such as build scripts, a
compiler, and libraries, and returns an installed package. As a pure function,
its result depends solely on its inputs—for instance, it cannot refer to
software or scripts that were not explicitly passed as inputs. A build function
always produces the same result when passed a given set of inputs. It cannot
alter the system’s environment in any way; for instance, it cannot create,
modify, or delete files outside of its build and installation directories. This
is achieved by running build processes in isolated environments (or containers),
where only their explicit inputs are visible.The result of package build functions is cached in the file system, in a special
directory called the store (see The Store). Each package is installed in a
directory of its own, in the store—by default under /gnu/store. The directory
name contains a hash of all the inputs used to build that package; thus,
changing an input yields a different directory name.
Packages are installed in /gnu/store, and symbolic links are created from the user’s “profile” to files in /gnu/store:
$ ls -l ~/.guix-profile/bin/python3 /home/cyril/.guix-profile/bin/python3 -> \ /gnu/store/gkx1f5wzlm5gn2iq7zlhpq5dziqvy768-python-3.5.0/bin/python3
Note that “gkx1f5wzlm5gn2iq7zlhpq5dziqvy768” is the hash of all the inputs of the package.
Thanks to this design, it is possible to install multiple versions of a package.
Guix environment, a “cross-language virtualenv”
Guix provides a “guix environment”
command that can create a trashable environment where only a given set of packages are installed. This is similar to virtualenv.
Let’s say we do not have the mox3 library installed on our system, and that we’d like to create an environment where it is available. We can use “guix environment” like this:
# mox3 is not available $ python -c "import mox3" Traceback (most recent call last): File "", line 1, in ImportError: No module named mox3 $ guix environment --ad-hoc python python-mox3 $ python -c "import mox3" $ ^D # Leave the environment
Note: Without “–ad-hoc”, this command would spawn an environment where all the dependencies of both “python” and “python-mox3” are available. With the “–ad-hoc” option, it makes sure “python” and “python-mox3” (and not their depencies) are available in the environment. In both cases, the packages are build in /gnu/store, but not installed in the user’s profile ($HOME/.guix-profile). The PATH and PYTHONPATH environment variables are also properly set up.
We could also ask Guix to run a given command in the environment:
$ guix environment --ad-hoc python python-mox3 --exec="python -c 'import mox3'" $
One can obviously create an environment with any of the packages provided by Guix.
How Guix may solve the issues encountered with virtualenv
One of the main reasons to use virtualenv is that it allows developers to install multiple versions of a given package, which is not always possible or easy with some package managers provided by GNU/Linux distributions. Thanks to its design, Guix allows this.
Guix can handle packages that are not on PyPI, like compilers, external libraries, packages written in another language… There is no need to install some of the required packages using the distribution’s package manager.
Guix provides a better isolation from the rest of the system. For instance, even if we have “git” installed, it becomes unavailable inside the spawned environment:
$ which git /usr/bin/git # Execute "which git" in a spawned environment where only "which" is installed # Note: "--pure" cleans environment variables. $ guix environment --ad-hoc --pure which --exec="which git" which: no git in (/gnu/store/scra07w6z2hnmackqj851rqznd7sznvm-which-2.21/bin)
The whole filesystem can still be accessed, though:
$ guix environment --ad-hoc --pure coreutils grep --exec="ls /usr/bin/git" /usr/bin/git
This is an issue, but soon enough, “guix environment” will be able to spawn a
container, so this will no longer be an issue.
Replacing virtualenv with guix in tox
The tox user interface is very handy, and developers are used to it, so we would like to keep it while managing packages using Guix rather than virtualenv.
A (quick) dive into Tox internals
Tox currently uses a “VirtualEnv” class to take care of all operations related to package management. Guix-tox simply adds a “GuixEnv” class that can be used as a drop-in replacement and makes the right calls to “guix environment” when needed.
By default, guix-tox behaves exactly like tox and manages virtual environments. But by passing the “–env=guix” option, developers get to drop virtualenv and run “guix environment” instead. You may find more details by checking the latest commits in the “guix-tox” branch of this repository.
Testing it
Let us test this new tool by running the test suite of python-keystoneclient with Python 2.7:
$ guix-tox --env=guix -epy34 ... File "/gnu/store/y5x6c38fzrbfl80jxrgjd6py2k88x12a-python-3.4.3/lib/python3.4/subprocess.py", line 1457, in _execute_child raise child_exception_type(errno_num, err_msg) FileNotFoundError: [Errno 2] No such file or directory: 'openssl' ...
Here, we see that the tests fail because at some point, they try to call the “openssl” binary, which is not part of the spawned environment, and is in no way specified in the requirements of python-keystoneclient. We can ask tox to pass this unexpected dependency to “guix environment” like this:
$ GUIX_TOX_EXTRA=openssl guix-tox --env=guix -epy34 py34 runtests: commands[0] | python setup.py testr --slowest --testr-args= py34 Guix env: guix environment --ad-hoc --pure python-pbr python-babel python-iso8601 python-debtcollector python-netaddr python-oslo.config python-oslo.i18n python-oslo.serialization python-oslo.utils python-prettytable python-requests python-six python-stevedore python-hacking python-coverage python-discover python-fixtures python-keyring python-lxml python-mock python-oauthlib python-oslosphinx python-oslotest python-pycrypto python-requests-mock python-sphinx python-tempest-lib python-testrepository python-testresources python-testtools python-webob python-bandit python-setuptools openssl python --exec=PYTHON=python3.4 python3.4 setup.py testr --slowest --testr-args= Ran 1133 (+1132) tests in 30.764s PASSED (id=65, skips=4)
Note that the guix command is printed out for debugging purposes.
Drawbacks
Fewer packages than on PyPI
Not all the packages available on PyPI may be packaged in Guix; they may also not always be up to date. This might not be such a big deal, since the most popular packages on PyPI might be enough for most applications. Also, users can write their own package definitions, possibly semi-automatically generated from PyPI or update existing packages. For instance, Guix currently only has python-babel 1.3, which is a bit old: the latest version is 2.0. One may write the following definition:
$ cat babel.scm (use-modules (guix packages) (guix download) (gnu packages) (gnu packages python)) (package (inherit python-babel) (version "2.0") (source (origin (method url-fetch) (uri (string-append "https://pypi.python.org/packages/source/B/Babel/Babel-" version ".tar.gz")) (sha256 (base32 "06zcwwfz3r04k6i9qr8j3j9nlljij67adv2pk2pnac0jj7qqv624")))))
And install python-babel 2.0 by running:
$ guix package -f babel.scm
We can see that only python-babel 1.3 is available in Guix, but that python-babel 2.0 is now installed in the user’s profile:
# List available packages that match "python-babel" $ guix package -A python-babel python-babel 1.3 out gnu/packages/python.scm:528:2 # List installed packages that match "python-babel" $ guix package -I python-babel python-babel 2.0 out /gnu/store/s0r0rv72ic2nww8j6672khqz1cc086b7-python-babel-2.0
Also, the guix refresh command may be updated in the future to handle packages from PyPI, and provide Guix developers with a way of easily keeping track of packages to upgrade.
Not all features are supported
Currently, only Python 2.7 and 3.4 are supported, since they are the only packaged versions in Guix. Building extensions “on the fly” in setup.py is not supported in guix-tox. Some other features might be missing. Should a failure happen, a developer could always remove the “–env=guix” (or use “–env=virtualenv”) and get the usual behaviour of tox.
Guix-tox is alpha software
Guix-tox currently only has a single user, and has not been thoroughly tested. But you can join the fun!
See also
- Ruby on Guix, by David Thompson, a similar approach for Ruby code
Hey,
cool stuff. I’m sad we can’t share these tools (in Nix community).
BTW, I’m packaging OpenStack Liberty for Nix (https://github.com/NixOS/nixpkgs/pull/10399) and if you want to chat about that, ping iElectric @ FreeNode #nixos
Hey!
I think it would be really easy to add support for “nix-shell” in tox. Nice to know you are packaging OpenStack for Nix. I might be interested in doing the same for GNU Guix (there are already some OpenStack libraries packaged).
[…] Guix-Tox: A Functional Version of Tox. […]