Status of Python 3 in OpenStack Mitaka

Now that most OpenStack services have reached feature freeze for the Mitaka cycle (November 2015-April 2016), it’s time to look back on the progress made for Python 3 support.

Previous status update: Python 3 Status in OpenStack Liberty (September 2015).

Services ported to Python 3

13 services were ported to Python 3 during the Mitaka cycle:

  • Cinder
  • Congress
  • Designate
  • Glance
  • Heat
  • Horizon
  • Manila
  • Mistral
  • Octavia
  • Searchlight
  • Solum
  • Watcher
  • Zaqar

Red Hat contributed to the Cinder, Designate, Glance and Horizon service porting efforts.

“Ported to Python 3” means that all unit tests pass on Python 3.4 which is verified by a voting gate job. It is not enough to run applications in production with Python 3. Integration and functional tests are not run on Python 3 yet. See the section dedicated to these tests below.

See the Python 3 wiki page for the current status of the OpenStack port to Python 3; especially the list of services ported to Python 3.

Services not ported yet

It’s become easier to list services which are not compatible with Python 3 than listing services already ported to Python 3!

9 services still need to be ported:

  • Work-in-progress:
    • Magnum: 83% (959 unit tests/1,161)
    • Cue: 81% (208 unit tests/257)
    • Nova: 74% (10,859 unit tests/14,726)
    • Barbican: 34% (392 unit tests/1168)
    • Murano: 29% (133 unit tests/455)
    • Keystone: 27% (1200 unit tests/4455)
    • Swift: 0% (3 unit tests/4,435)
    • Neutron-LBaaS: 0% (1 unit test/806)
  • Port not started yet:
    • Trove: no python34 gate

Red Hat contributed Python 3 patches to Cue, Neutron-LBaaS, Swift and Trove during the Mitaka cycle.

Trove developers are ready to start the port at the beginning of the next cycle (Newton). The py34 test environment was blocked by the MySQL-Python dependency (it was not possible to build the test environment), but this dependency is now skipped on Python 3. Later, it will be replaced with PyMySQL on Python 2 and Python 3.

Python 3 issues in Eventlet

Four Python 3 issues were fixed in Eventlet:

Next Milestone: Functional and integration tests

The next major milestone will be to run functional and integration tests on Python 3.

  • functional tests are restricted to one component (ex: only Glance)
  • integration tests, like Tempest, test the integration of multiple components

It is now possible to install some packages on Python 3 in DevStack using USE_PYTHON3 and PYTHON3_VERSION variables: Enable optional Python 3 support. It means that it is possible to run tests with some services running on Python 3, and the remaining services on Python 2.

The port to Python 3 of Glance, Heat and Neutron functional and integration tests have already started.

For Glance, 159 functional tests already pass on Python 3.4.

Heat:

Neutron: the Add the functional-py34 and dsvm-functional-py34 targets to tox.ini change was merged, but a gate job hasn’t been added for it yet.

Another pending project is to fix issues specific to Python 3.5, but the gate doesn’t use Python 3.5 yet. There are some minor issues, probably easy to fix.

How to port remaining code?

The Python 3 wiki page contains a lot of information about adding Python 3 support to Python 2 code.

Join us in the #openstack-python3 IRC channel on Freenode to discuss Python 3!

Python 3 Status in OpenStack Liberty

The Python 3 support in OpenStack Liberty made huge visible progress. Blocking libraries have been ported. Six OpenStack applications are now compatible with Python3: Aodh, Ceilometer, Gnocchi, Ironic, Rally and Sahara. Thanks to voting python34 check jobs, the Python 3 support can only increase, Python 2 only code cannot be reintroduced by mistake in tested code.

Progress made in OpenStack Juno and Kilo

The OpenStack port to Python 3 started slowly in 2013 during the development of OpenStack Havana. For the previous Python 3 status, read again Status of the OpenStack port to Python 3 (Cyril Roelandt, February 2014), see also: Why should OpenStack move to Python 3 right now? (Victor Stinner, December 2013).

The year 2013 was focused on porting third-party libraries and Oslo Incubator. The year 2014 was more focused on porting OpenStack clients and OpenStack common libraries. During the year 2015, we reached the interesting point: start porting OpenStack libraries and OpenStack applications. The Moving Apps to Python 3 etherpad prepared the work for the Liberty cycle.

While Python 3.3 was targeted in 2014, the new Python target version is 3.4. In parallel, Python 2.6 was dropped in more and more OpenStack client and libraries which also simplifies the port to Python 3.

setuptools, pbr and environment markers

It was not possible to have dependencies specific to Python 2 and others specific to Python 3. While dependencies like eventlet or MySQL-Python were incompatible with Python 3, it was annoying to run tests, even manually. Some dependencies were also installed on Python 2.7 whereas they were only neeeded on Python 2.6.

A first step (workaround) was to support requirements-py2.txt and requirements-py3.txt for dependencies specific to Python 2 / Python 3, and test-requirements-py2.txt and test-requirements-py3.txt for test requirements.

A lot of work was done in pip (ex: support environment markers in requirements) and pbr to support environment markers. Environment markers come from the PEP 426 (Metadata for Python Software Packages 2.0): see the Environment markers section. Examples of environment markers: sys_platform == 'win32' (detect Windows) and python_version=='2.7' (Python 2.7).

requirements-py2.txt and requirements-py3.txt were merged again into one unique requirements.txt file, and same for test requirements. The advantage of having environment markers is that it becomes possible to publish a universal wheel package which has different dependencies depending on the Python version.

OpenStack Common Libraries

Before Liberty, the port of almost all OpenStack applications was blocked by dependencies, OpenStack libraries and third-party libraries incompatible with Python 3. First, the root project oslo-incubator project was slowly ported to Python 3. In the meanwhile, this project was splitted into real libraries like oslo.config and oslo.utils. When a new library was created, we tried to ensure that the first release directly supports Python 3.

eventlet

The eventlet library is used by all OpenStack applications, but it was not compatible with Python 3. Hopefully most OpenStack clients don’t use eventlet and so could be ported to Python 3.

The eventlet project was slowly ported since 2014.  In 2015, it’s now fully compatible with Python 3, since the version 0.17.3.

MySQL

The mysql-python library also blocked almost all OpenStack applications.

We tried porting it to Python 3 at first, but the maintainers were not really reactive, so we thought it might be easier to just replace it with another library.

There was a discussion to replace mysql-python with mysqlclient, a fork adding Python 3 support, fixing various bugs and adding some features.

It was decided to replace mysql-python with PyMySQL instead. PyMySQL is a completely new MySQL driver written fully in Python. The main advantage is that it can be monkey-patched by eventlet to become asynchronous.

OpenStack libraries and other libraries

We ported or helped to port the following third-party libraries to Python 3 because they blocked OpenStack applications. We also pushed for releases. We also ported “OpenStack libraries”: libraries written for a specific application, like glance_store for Glance. Releases including Python 3 fixes:

  • ecdsa 0.10
  • glance_store 0.7.0: library written for Glance
  • netifaces 0.10.4
  • nose-exclude 0.4: blocked Horizon unit tests
  • os-brick 0.3.2: library written for Cinder
  • PyEClib 1.0.9
  • python-memcached 1.56: blocked keystone middleware (so all applications)
  • routes 2.2: blocked many applications (cinder, glance, keystone, neutron, etc.)
  • websockify 0.7.0

suds was replaced with suds-jurko: suds blocked cinder, oslo.vmware and nova. suds is no more maintained, whereas suds-jurko is maintained and supports Python 3.

Python 3 Status in OpenStack Liberty

Python 3 Status of OpenStack Common Libraries

All OpenStack common libraries are now compatible with Python 3: the entire test suite can run with Python 3.4. There are currently 20 libraries:

  • cliff
  • oslo.concurrency
  • oslo-incubator
  • oslo.config
  • oslo.context
  • oslo.db
  • oslo.i18n
  • oslo.log
  • oslo.messaging (*)
  • oslo.middleware
  • oslo.rootwrap
  • oslo.serialization
  • oslosphinx
  • oslotest
  • oslo.versionedobjects
  • oslo.vmware
  • oslo.utils
  • pylockfile
  • stevedore
  • taskflow

(*) The Qpid and AMQP transports of oslo.messaging are not yet compatible with Python 3. The legacy Qpid driver is deprecated, buggy and unmaintained. The AMQP driver is being ported to Python 3, it’s almost done.

Python 3 Status of OpenStack Clients

17 clients are fully compatible with Python 3, the entire test suite can run with Python 3.4:

  • keystonemiddleware
  • python-barbicanclient
  • python-ceilometerclient
  • python-cinderclient
  • python-glanceclient
  • python-heatclient
  • python-ironicclient
  • python-keystoneclient
  • python-manilaclient
  • python-marconiclient
  • python-neutronclient
  • python-novaclient
  • python-openstackclient
  • python-saharaclient
  • python-swiftclient
  • python-troveclient
  • python-tuskarclient

2 clients are being ported to Python 3:

  • python-designateclient
  • python-fuelclient (StackForge project)

Python 3 Status of OpenStack Applications

6 applications are already fully compatible with Python 3! All unit tests pass on Python 3.4. Congratulations to their maintainers.

  • Aodh
  • Ceilometer
  • Gnocchi
  • Ironic
  • Rally
  • Sahara

6 applications are being ported to Python 3 with a voting python34 check job to avoid Python 3 regressions. To be able to port the code incrementally, a subset of tests are run on Python 3 to have a working check job. More and more tests are added to the subset each time that more code is ported to Python3. Applications partially ported:

  • Neutron: 99.9% ! (7541 tests/7551, only 10 tests remain)
  • Heat: 86% (4386 tests/5119)
  • Cinder: 38% (2532 tests/6659)
  • Nova: 19% (2746 tests/14358)
  • Glance: 18% (512 tests/2779)
  • Horizon: 15% (238 tests/1605)
  • Keystone: 12% (524 tests/4318)

Keystone is still blocked by python-ldap and ldaptool. python-ldap may be replaced with pyldap, a fork compatible with Python 3.

3 applications are still at an early stage of the Python 3 port:

  • Designate
  • Manila
  • Swift

No more Python 3 regressions

In previous cycles, it was common to reintroduce code incompatible with Python 3 in files which were already ported to Python 3. We tried to avoid this issue by running unit tests on Python 3 on voting gates. When a test is run on Python 3, Python 2 only code cannot by introduced anymore in the tested code.

Some projects like Nova load more tests than the subset of tests executed on Python 3, to prevent people from adding Python 3 incompatible syntax and imports.

The Python 3 support is now also required to add a new dependency to OpenStack global requirements.

In general, more and more developers are aware of Python 3 and take it into account in their development.

What’s Next? How can you help?

Since most unit tests of Neutron and Heat are already ported to Python 3, we can expect a full support during the next Mitaka cycle. The port of other applications will also continue in parallel.

The next major task is to run functional tests on Python 3. Doug Hellmann wrote a specification Enabling Python 3 for Application Integration Tests. The specification was approved by the PTLs of the different projects. Now it’s time to implement the specification. The first step will be to add an option to DevStack to run clients and some applications on Python 3 (while other applications will still run on Python 2).

You can help to port OpenStack to Python 3 by reviewing patches, writing new patches and testing clients and applications on Python 3. The Python 3 wiki page is the central place to collaborate on this project.

Thanks

We would like to thank:

  • Andrey Kurilin (Rally)
  • Cyril Roelandt (Neutron)
  • Davanum Srinivas aka dims (Nova)
  • David Stanek (Keystone)
  • George Peristerakis (Horizon)
  • Ihar Hrachyshka (Neutron)
  • Jaivish Kothari aka janonymous (Swift)
  • Jeremy Stanley who coordinated efforts on switching from MySQL-python to PyMySQL
  • Pradeep Kumar Singh (Designate)
  • Sirushti Murugesan (Heat)
  • Victor Sergeyev (Ironic)
  • Victor Stinner aka haypo (Aodh, Ceilometer, Cinder, Glance, Horizon, Nova, Swift)

There are much more developers involved in the Python 3 porting effort. Thanks to all of them!

This article was written by Cyril Roelandt and Victor Stinner.

Use the new asyncio module and Trollius in OpenStack

Asynchronous programming is hard. In the past, the Nova project used Tornado, then Twisted and it is now using eventlet which also became the defacto standard in OpenStack. Eventlet is not perfect, we will explain why we consider that eventlet has major flaws. We will then introduce the asyncio module of Python 3.4, how we plan to use it in OpenStack, to finish with the current status of this integration.

What’s wrong with eventlet?

Eventlet is “a concurrent networking library for Python that allows you to change how you run your code, not how you write it” according to its documentation.

The eventlet module has two major issues: debugging is hard because of the implicit task switching and it does not support Python 3 (see the Support Python 3.3 issue). The eventlet dependency blocks at least the porting to Python 3 of 9 OpenStack servers (ceilometer, cinder, glance, heat, horizon, keystone, neutron, nova, swift).

In OpenStack, eventlet is used with monkey-patching enabled. Monkey-patching the Python standard libraries replaces blocking functions with asynchronous functions using greenthread. For example, the Python time.sleep() function which “suspends execution for the given number of seconds” is replaced with greenthread.sleep which “yields control to another eligible coroutine until at least seconds have elapsed”.

The problem is that it becomes hard to guess if a function may or may not switch to another task (“green thread”). Or worse, the function may switch in a new version of a module, you won’t notice the change. The developer must be aware of that and write the code with this fact in mind. This has led to real bugs, remember this OpenStack reaction: when adding sleep(0) fixes an eventlet test. 🙂 Just one example of eventlet.sleep(0): Sleep to simulate concurrency and allow other threads to work in Ceilometer tests (most concurrency tests of Nova use eventlet.sleep).

Eventlet also monkey-patches the threading module so it is not possible to use operating system threads, green threads are used everywhere.

Eventlet relies on the greenlet module to manage green threads. The greenlet module is not portable: it uses assembler code with one version per architecture and per platform to switch between green threads. Greenlet contains code specific to CPython and so cannot be used with Jython or IronPython. PyPy supports greenlet since PyPy 2.0. Even if “it works”, eventlet will not be integrated in CPython, it is difficult to maintain and so is not a sustainable solution.

The new asyncio module (PEP 3156) and the Trollius project

For a nice introduction to asyncio, read the report of Guido Van Rossum’s keynote at Pycon US 2013: PyCon: Asynchronous I/O by Jake Edge (March, 2013). The development started outside Python in a project called Tulip for Python 3.3, the name of the Python module is “asyncio” (use import asyncio in Python).

The asyncio module has been designed as a superset of existing libraries: Twisted, Tornado, gevent, eventlet, etc. The design is the PEP 3156Asynchronous IO Support Rebooted: the asyncio Module” written by Guido van Rossum, author of the Python language. The PEP has been written with the help of developers of the existing libraries and the design reuses good ideas of existing libraries, like transports and protocols of Twisted. Asyncio offers a nice syntax for coroutines using the new yield from syntax of Python 3.3: PEP 380 “Syntax for Delegating to a Subgenerator”.

The asyncio module is written in pure Python, and therefore portable. It is built on top of existing modules like concurrent.futures, subprocess, threading and the new selectors module (which is itself built on top on the select module). asyncio is a glue between modules of the Python standard library and existing asynchronous frameworks, but it can be used alone. It also adds new features like tasks using coroutines. It supports blocking functions as well with its executor API which uses a pool of threads by default. It uses operating system threads which can run in parallel (especially when executing C functions releasing the GIL). The default executor can be replaced to use a pool of processes, or a custom executor.

PEP 3156 has been accepted and asyncio is part of the Python 3.4 standard library. Asyncio is now very well tested by a farm of more than 33 buildbots on various architectures (x86, x86_64, PPC64, ARM7, SPARC) and operating systems (Linux, Windows 2003/XP/7, Mac OS X 10.4/10.6, FreeBSD 6/7/9/10, OpenBSD, Solaris 10, OpenIndiana). Enovance ported asyncio to Python 2.6 and 2.7 as the new Trollius project: Trollius works on Python 2.6-3.4. We also wrote the documentation for the asyncio module: asyncio – Asynchronous I/O, event loop, coroutines and tasks.

Even before the official release of Python 3.4 (scheduled for March 16th), there are already many projects supporting asyncio. There are asyncio event loops for greenlet, gevent, libuv, GLib, Tornado and 0MQ. There are database drivers for PostgreSQL, Redis, MongoDB and memcached. There are HTTP clients and servers, web sockets, and a gunicorn worker. See asyncio third party for the whole list, some of them are still experimental.

For more information on asyncio, see my notes about asyncio: talks about asyncio, Tulip backends and event loops, Tulip projects, etc. For a more technical comparison of the different asynchronous I/O models in Python, read the nice Async I/O and Python article by Mark McLoughlin (June, 2013).

Differences between Trollius and Tulip

The Trollius project is based on Tulip: in the Mercurial repository, it is a branch based on the Tulip branch, so Trollius can easily merge new changes from Tulip. We are working directly on asyncio and update the Trollius project regularly to keep it up to date.

The major difference between Tulip and Trollius is the syntax of coroutines: Trollius uses yield From(...) instead of yield from ..., and raise Return(x) instead of return x. Except for the syntax of coroutines, the API is the same: classes, methods, functions, etc. It is possible to use the same code base for Trollius and Tulip using callbacks. For example, this solution is used by the AutobahnPython project which also supports Twisted.

Also read the Trollius documentation.

Plan to use asyncio in OpenStack

Straightforward dropping of Python 2 support and replacing eventlet with asyncio is not possible: there are a lot of existing OpenStack setup running on Python 2. The Tulip project cannot be used in OpenStack, because the porting of OpenStack to Python 3 is still in progress. Trollius is a more realistic option because it works on Python 2 and 3.

There are different options to use asyncio in OpenStack:

  • Replace eventlet completely with Trollius
  • Plug Trollius into eventlet: eventlet would execute Trollius tasks; eventlet would be the Trollius event loop
  • Plug eventlet into Trollius: Trollius would execute eventlet tasks; Trollius would be an eventlet hub

Replacing eventlet with Trollius at once is not possible, it’s not how OpenStack is developed. All changes in OpenStack are carefully reviewed, well tested, and usually very small. The development is incremental.

Plugging Trollius into eventlet can be implemented using the greenio project which is an asyncio event loop using greenlet. It doesn’t look to be the preferred option of the OpenStack developers.

Plugging eventlet into Trollius can be implemented using the eventlet hub API. We are working on a proof-of-concept. This way would be a smooth transition to asyncio. Existing code would continue to use eventlet, and new code can directly use the asyncio API.

Also read recent threads on the openstack-dev mailing list about asynchronous programming, asyncio and Python 3:

Status of asyncio in OpenStack

The first step was to add Trollius to the global-requirements.txt: the patch has been merged.

The Oslo Messaging project was selected to experiment asyncio because it was designed to support different asynchronous libraries. Oslo Messaging has an “executor API” with two available implementations: blocking and eventlet. The blueprint Oslo/blueprints/asyncio proposes to add a new Trollius executor, implementation: Add a new asynchronous executor based on Trollius.

The next step is to write an experimental eventlet hub on Trollius.

Why should OpenStack move to Python 3 right now?

Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.

This article describes Python 3.3 with a preview of Python 3.4 which is scheduled at the end of february 2014!

New Python 3 features

Python 3 has too many useful new features to list them all here. Read each “What’s New in Python 3.x” document for the full list of changes: Python 3.0, Python 3.1, Python 3.2, Python 3.3 and Python 3.4. A lot of exciting stuff (asyncio) is coming in Python 3.4!

To give you an overview of new features, 8 new modules were added between Python 2.7 and 3.3:

  • concurrent.futures (3.2): high-level interface for asynchronously executing callables, pool of threads and pool of processes ;
  • faulthandler (3.3): debug module to dump the traceback of Python threads on a crash, on a signal or after a timeout ;
  • importlib (3.1): portable implementation of the import statement written in pure Python (a minor subset is also available in Python 2.7) ;
  • ipaddress (3.3): create, manipulate and operate on IPv4 and IPv6 addresses and networks ;
  • lzma (3.3): classes and convenience functions for compressing and decompressing data using the LZMA compression algorithm ;
  • tkinter.ttk (3.1): Tk themed widget set introduced in Tk 8.5 ;
  • unittest.mock (3.3): replace parts of your system under test with mock objects and make assertions about how they have been used ;
  • venv (3.3): create lightweight “virtual environments” with their own site directories.

Python 3.2 has also a new argparse module: easy to write user-friendly command-line interfaces, but this module is also available in Python 2.7.

As many modules have been added between Python 3.3 and 3.4 as between Python 2.7 and Python 3.3! Python 3.4 has 7 new modules:

  • asyncio: infrastructure for writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives ;
  • enum: set of symbolic names (members) bound to unique, constant values ;
  • ensurepip: bootstrap the pip installer into an existing Python installation or virtual environment ;
  • pathlib: classes representing filesystem paths with semantics appropriate for different operating systems ;
  • selectors: high-level and efficient I/O multiplexing, built upon the select module primitives ;
  • statistics: functions for calculating mathematical statistics of numeric (Real-valued) data ;
  • tracemalloc: debug tool to trace memory blocks allocated by Python.

Having more modules in the Python standard library reduces the number of dependencies and so simplify the deployment of an application.

Finally, Python 3 has a much better support of Unicode. In short, it makes the internationalization (i18n) of an application easier. When porting an existing application written for Python 2 to Python 3, you may have the opposite feeling because all Unicode related issues should be fixed at once. If you write a new Python 3 application from scratch, Unicode “just works” and you don’t have to care of all these bytes versus characters, encodings and mojibake issues.

For a nice overview of changes between Python 2.7 and 3.3, see the “Python 3.3: Trust Me, It’s Better Than Python 2.7” talk of Dr. Brett Cannon at PyCon US 2013: slides and video.

Python 3 is fast

Examples of Python 3 optimizations:

  • The mechanism for serializing execution of concurrently running Python threads (generally known as the GIL or Global Interpreter Lock) has been rewritten. Among the objectives were more predictable switching intervals and reduced overhead due to lock contention and the number of ensuing system calls.
  • range(), map(), dict.keys(), etc. now create an iterator or a generator instead of a temporary list to use less memory, it can also be faster
  • Unicode strings always use the most compact storage, up to 4 times smaller (ex: 1 byte per ASCII character instead of 4). A nice side effect is that some string operations on ASCII strings are up to 4 times faster. Compared to Python 2.7, Python 3.3 uses a little bit less memory than Python 2.7 on Django, see PEP 393: Performance and resource usage.
  • The decimal module has been reimplemented in C: it is now between 12x and 120x faster
  • The main loop evaluating bytecode is now 20% faster thanks to “computed goto
  • The Python peephole optimizer produces more efficient bytecode. None, False and True are now keywords and so can be optimized. “x in {1, 2, 3}” pattern is optimized as “x in frozenset({1, 2, 3})” where the frozenset is stored as a pre-built constant.
  • Common text codecs (ASCII, Latin1, UTF-8) are two to four times faster.
  • The json module now has a C extension to substantially improve its performance.

Other Python 3 changes made Python 3 slower on some operations. To be fair, overall performances of Python 3.3 are almost the same than Python 2.7 performances. But some functions of your application can now be much faster on Python 3.

See the “Optimizations” section of each “What’s New in Python 3.x” document for the full list of optimizations: Python 3.1, Python 3.2, Python 3.3 and Python 3.4.

Status of Python 3 in Linux distributions

All major Linux distributions provide Python 3.3, or Python 3.2. RedHat 6 has been providing Python 3.3 in its new “Red Hat Software Collections” since September 2013.

ArchLinux already switched to Python 3 by default three years ago. Fedora and Ubuntu plan to switch in a near future. Ubuntu wants to go further, remove Python 2 from the default installation in april 2014: Shipping only Python 3 on the 14.04 CD. Fedora scheduled the switch in Fedora 22 (december 2014): Python 3 as the Default Implementation.

No more new features in Python 2.7

In Python, only one branch accepts new features: the default branch. Currently, the default branch is the future Python 3.4 release. Python 2.7 only accepts bugfixes, no more new features nor syntax changes. Read the “Python 2.8 Un-release Schedule” (PEP 404) for the rationale.

The development branches of Python 2 and 3 diverged so much in 5 years that it would require too much work to fix some bugs in Python 2. Python developers don’t want to duplicate their effort and prefer to focus on the next release. Another reason is to not introduce regressions. Python 2.7 is now very stable and heavily used in production. A single minor change might introduce a regression, even if Python has a very good code coverage with its huge test suite.

A recent example of a bug that cannot be fixed in Python 2.7 is the “Secure and interchangeable hash algorithm” (PEP 456) which fixes a security vulnerability (“hash DoS”). The vulnerability was partially fixed in Python 2.7: the hash function can now be randomized, but it should be done explicitly on the command line or using an environment variable. Read the Denial of service via hash collisions article (Jake Edge, January 2012) for more information. In Python 3.3, the hash function is randomized by default. Python 3.4 will use a new fast cryptographic hash function “SipHash”, see Python adopts SipHash (Jake Edge, November 2013). Python 2.7 and 3.3 are vulnerable but will not be fixed: see the end of the Python issue #14621: “Hash function is not randomized properly”.

Another example of a bug that cannot be fixed in Python 2.7 is the “Safe object finalization” (PEP 442). This PEP fixes an annoying memory leak of Python, which existed since the first Python version (23 years ago). When a group of Python objects are linked all together and at least one object has a destructor (__del__ method), these objects will never be deleted. It is difficult to identify reference cycles, but it is possible to workaround this issue using weak references. Thanks to the PEP 442, implemented in Python 3.4, objects with a destructor are now deleted by the garbage collector.

Why should OpenStack move to Python 3 right now?

The motivation to move away from Python 2 is the technical debt. OpenStack is one of the largest opensource project in term of lines of code, more than 2.5 million lines of code. If an application is not updated to track evolution of new features, it dies slowly. A technical debt has a concrete price: the longer you wait to port OpenStack to Python 3, the more expensive the portage will be.

Linux distributions want to slowly remove Python 2 and stop supporting it. Python 2.7 has more and more bugs, some bugs cannot be fixed anymore. If OpenStack is not ported on Python 3, it will more and more difficult to maintain it in a near future.

The development on Python 2 becomes also more expensive because new useful features must be backported. The backported code should be maintained, whereas new modules part of the Python standard library are maintainted by Python developers.

Switching to Python 3 will also improve performances, reduce the number of dependencies, and more generally to make OpenStack more robust.

The portage of OpenStack to Python 3 already started. A second blog article will give a status of this portage and propose a planning to port the servers, not only the clients.