pbauer

Towards Plone 6

A report from the Alpine City Sprint in Innsbruck

Two of the main items on the Plone roadmap are the move to Python 3 and a more active role in the development of the underlying stack of Plone, namely Zope. After the recent uptake of activity in development of Zope (see here and here) it was a sensible first task to try to get Plone to run on the most recent Zope stack. What exactly does that mean?

There is a PLIP that proposes to upgrade Plone's Zope-related dependencies and is approval by the Plone Framework Team to be a part of Plone 6. As with all PLIPs, the point is not only to agree on a task and a implementation but also actually doing the work. After two smaller efforts at sprints last year where only 2 or 3 people were working on the task we finally got a critical mass of developers together for the Alpine City Sprint. 15 people sprinted and worked on this one task.

Off to a good start

On the day before the sprint I rebased some of our branches and with a couple of quick and really dirty fixes I got Plone run on Zope 4. Many features were still not working but a working instance certainly provided motivation for this very dedicated sprint.

WSGI replaces the ZServer

The ZServer is now a optional dependency and the new WSGI-Publisher is the default publisher making Plone fully WSGI-compliant! Thomas Schorr added some necessary buildout-parts to run Plone on uwsgi, waitress and gunicorn. Documentation is here and examples are here. Even the testbrowser used for all of Plone's functional tests now uses the WSGI-Publisher. The advantages of WSGI are obvious: It makes it easier to have interact Plone with other services and some features can be developed as WSGI-middleware. It'll be interesting to see if we will make more use of WSGI for Plone in the future than "simply" for serving.

ZCatalog 4 and ZODB 5

The result of the sprint is beyond what was originally proposed in the PLIP. We now not only use the master branch from Zope and the zope-toolkit packages (zope.interface, zope.component etc.) but also ZODB 5 and ZCatalog 4.

One change that required some work was that ZCatalog, as it now returns a empty list when you pass no query or a invalid query. Especially in the cases when you query the catalog for a non-existing index (portal_typos='foo') getting all brains from the Catalog seems like a bad idea. The down-side that change there is that there is no obvious way to walk through all brains in the catalog any more. Johannes Raggam added the convenience-method catalog_get_all for this.

The changes in ZODB are numerous but for Plone-developers nothing much really changes except that is supposed to be faster and supports Python >= 3.3 and 2.7. For the future it will be also interesting to discuss if switching to NewtDB would be a option. NewtDB runs ZODB 5 with RelStorage and stores data as json in Postgres.

Python 3

The sprint did not try to move Plone to Python 3 yet but rather clear the way so we can do it. The last remaining blocker with C-code was RestrictedPython and Alexander Loechel and Michael Howitz continued working on the rewrite based on the Python AST module. At the end of the sprint they were happy to announce that it is no longer a blocker for our move to Python 3. All the other packages that rely on C-code that seems mysterious to mere Python-mortals (ExtensionClass, AccessControl, Acquisition, Persistence) were already ported to Python 3. RestrictedPython is a very useful tool that allows you to execute untrusted Python code that is loaded within a running Python program. It checks and modifies source code to only allow the execution of a restricted subset of Python.

What remains to be done?

  • Fix some failing tests (DONE): As of today we are down to 8 failing tests 6 failing tests 2 failing tests 0 failing tests!!! 
  • Update tasks and tickets
  • Merge changes: A lot of the changes were already merged in the master-branches for the current coredev (for Plone 5.1). But some merges will have to wait until Plone 5.1 is released because they only target Plone 6.0. Once all test are green and all our zope-related branches are merged we can make a release of Zope 4.0a3 and switch from using source-checkouts for everything (auto-checkout = *) to using released versions.
  • Documentation: We need to document what changed in Zope from the perspective of Plone. Most notably some lesser features of the ZMI are gone. These and other changes need to go into the upgrade-guide for the next Plone-Version. Also the new setup for various WSGI-Servers needs to be well-documented so that developers can  copy&paste best-practice configs into their buildouts. The documentation was started in a Google Doc where everyone can contribute. Once it's stable we'll move it to their respective places.
  • Upgrades: So far it seems like the changes we made did not make any upgrade-steps necessary. After adding a quick hack to work around the missing zope-controlpanel I was able to upgrade a existing Plone 5.0.6 site without any issues.
  • Profiling: Initial results with from plone.instanceprofiling indicate the speed of Plone has increased but we need more real-life tests before a final judgement. It will also be interesting to see the impact of Python 3 on speed.

Inside the sprint

As usual during the sprint we ran into a bunch of blockers and hard nuts to crack. The hardest one was probably that the current users roles were mysteriously dropped during a request after running trough some C-code. Like the memory of a bad dream, the bug vanished in the light of the morning - actually someone switched to different branch of AccessControl. On the first day already, Matthew Wilkes had committed to a metaclass in Persistence (thankfully brown-bagging my terrible late-night hack and replacing it with something much more elegant), and installed gdb to debug the issue with the dropped roles.

Unfortunately many of our tests were broken since the testlayer did not use chameleon, thankfully after a lot of debugging the solution found by Thomas Schorr was pretty simple. Also tough was a issue that broke tests in weird ways. Gladly Jens Klein, our host, dug deep and found the culprit: Combining bundles modified the response's Content-Type header, surprise! A issue similar to https://github.com/plone/Products.CMFPlone/pull/1438.

Shops!

By the way: We also had a demo of bda.plone.shop and a discussion about shop-solutions for Plone. bda.plone.shop has plenty of great features and seems really well thought-through. It still lacks some end-user documentation with screenshots and a new release but we will certainly use it in our next project that involves a shop.

Where do we go from here?

To the pool in Sorrento! Plone Open Garden (PLOG) in beautiful Sorrento is the perfect opportunity to discussing the further roadmap to Python 3 and continue working on this. PLOG will have a strong focus on the headless CMS project for which a Python 3-compatible Plone 6 will be a mayor milestone. Register now and enjoy a week at the pool with the Plone community.

And then on to Finnland to the Midsummer Sprint! All the work on the backend needs to be balanced with some fun UX and UI work to enhance the content editors experience.

Summary

We worked on many important parts of the Plone Roadmap: Removing unused components, clean up the code base, moving to Python 3, and making Plone WSGI compliant. There is now a good chance that Plone will be on Python 3 this or next year. A great sprint, dedicated and smart people, good beer and food and nice mountains. Thanks to the organizers of Klein und Partner who worked hard to keep us fed, fueled and entertained and thanks to all the sprinters who dedicated so much of their time for such a great open source project!

Update (19.02.2017)

  1. Last week gocept announced there will be another Zope sprint in Mai with the goal to port Zope to Python 3. Already 11 developers signed up.
  2. David Glick fixed the remaining failing tests (mosty issues with the new handling of exceptions).