Blog

I got interviewed about Plone 5.2

erstellt von Philip Bauer zuletzt geändert: 2019-07-19T15:15:53+01:00
I got asked a bunch of more or less relevant questions regarding Plone, Python 3 and the future.

Click here to read the interview on plone.org.

Four options to try Plone 5.2 on Python 3

erstellt von Philip Bauer zuletzt geändert: 2019-03-08T08:40:10+01:00
Try these different options to get started with Plone on Python 3.

Demo pages

There are nightly build of the current Plone 5.2 coredev (the development version) with Python 2 and 3:

Minimal Buildout

Here is a minimal buildout to run Plone 5.2rc1 on Python 3:

[buildout]
parts = instance
extends = https://dist.plone.org/release/5.2rc1-pending/versions.cfg

[instance]
recipe = plone.recipe.zope2instance
eggs =
Plone
Pillow

You set it up like this:

$ python3.7 -m venv .
$ ./bin/pip install -r https://dist.plone.org/release/5.2rc1-pending/requirements.txt
$ ./bin/buildout

And start it as usual with ./bin/instance fg

Standalone Development buildout

[buildout]
extends = https://dist.plone.org/release/5.2rc1-pending/versions.cfg

parts =
    instance
    zopepy
    packages
    test
    robot

eggs =
    Plone
    Pillow
    collective.easyform

test-eggs =
    collective.easyform [test]

auto-checkout =
    collective.easyform

extensions =
    mr.developer

show-picked-versions = true

[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
eggs = ${buildout:eggs}
debug-mode = on
verbose-security = on

[zopepy]
recipe = zc.recipe.egg
eggs =
    ${buildout:eggs}
interpreter = zopepy
scripts =
    zopepy
    plone-compile-resources

[packages]
recipe = collective.recipe.omelette
ignore-develop = False
eggs = ${buildout:eggs}
ignores = roman

[test]
recipe = collective.xmltestreport
eggs = ${buildout:test-eggs}
defaults = ['--auto-color', '--auto-progress']

[robot]
recipe = zc.recipe.egg
eggs =
    ${buildout:test-eggs}
    Pillow
    plone.app.robotframework[reload,debug]

[sources]
collective.easyform = git git@github.com:collective/collective.easyform.git branch=python3

[versions]

Starzel buildout

The buildout that we at Starzel.de use supports Plone 5.2rc1 with Python 2 and 3.

https://github.com/starzel/buildout

It has some nice features:

  • It extends to config- and version-files on github shared by all projects that use the same version of Plone.
  • It allows to update a project simply by changing the version it extends.
  • It allows to update all projects of one version by changing remote files (very useful for HotFixes).
  • It is minimal work to setup a new project.
  • It has presets for development, testing, staging and production.
  • It has all the nice development-helpers we use.

Quickstart:

$ git clone -b 5.2rc1.x https://github.com/starzel/buildout <SOME_PROJECT>
$ cd <MY_PROJECT>

Remove all files that are not needed for a project but are only used for the buildout itself.

$ rm -rf linkto README.rst README.txt .travis.yml secret.cfg_tmpl VERSION.txt local_coredev.cfg CHANGES.rst

If you're not developing the buildout itself you want a create a new git repo.

$ rm -rf .git && git init

Add a file that contains a passwort. Do not use admin as a password in production!

$ echo -e "[buildout]\nlogin = admin\npassword = admin" > secret.cfg

Symlink to the file that best fits you local environment. At first that is usually development. Later you can use production or test. This buildout only uses local.cfg and ignores all local_*.cfg.

$ ln -s local_develop.cfg local.cfg

Create a virtualenv in Python 2.7 or Python 3.7 (Plone 5.2 only).

$ virtualenv .  # for Python 2.7
$ python3.7 -m venv .  # for Python 3 (Plone 5.2 only)

Install and configure Plone

$ ./bin/pip install -r requirements.txt
$ ./bin/buildout

Wrapup

I hope these options help you get started with Python 3. For serious projects you will likely create you own buildout.

Update (4.2.2019)

You should use the Plone coredev for developing Plone itself:

$ git clone git@github.com:plone/buildout.coredev.git coredev
$ cd coredev
$ git checkout 5.2
$ python3.7 -m venv .
$ ./bin/pip install -r requirements.txt
$ ./bin/buildout
$ ./bin/instance fg

Update 2 (24.2.2019)

Since Plone 5.2b2 wsgi is the default, even when running Python 2. See https://github.com/plone/Products.CMFPlone/issues/2763

This means you do not need to enable wsgi = on since it will be enabled automatically. You can still choose to switch wsgi = off in Python 2 if you have a good reason to do so (e.g. you need FTP or WebDAV). I updated the examples accordingly.

Update 3 (8.3.2019)

Release candidate 1 of Plone 5.2 is pending. See https://community.plone.org/t/plone-5-2rc1-soft-released/8163. I updated the examples accordingly.

Plone finally supports Python 3!

erstellt von Philip Bauer zuletzt geändert: 2018-11-29T22:41:55+01:00
Our favourite Open Source CMS has landed on Python 3. Here is what you need to know about migrating projects.

At the Plone Conference 2018 in Tokyo I gave a plenary talk in which I told the story how Plone got to support Python 3. Ten minutes before my talk Plone 5.2a1 was released. That release was the culmination of a 3-year long journey (or even 5 years if you start in Brazil). If you want to learn how we got there you should watch the talk.

Where are we now?

Plone 5.2 supports Python 3.7, 3.6 and 2.7. It is currently in alpha but the plan is to release a final version in February 2019 right after the Alpine City Sprint. You should seriously consider joining that sprint if you plan to migrate your own projects. Working together with the people who ported Plone to Python 3 will give you the know-how to succeed in your own migrations!

But in fact you should not wait until February to start working with Plone on Python 3 but start right now so you'll have enough time to migrate to Python 3 before January 1st 2020 (End of Life of Python 2).

To make testing and working with Python 3 easier some tools and addons were already working on Python 3.

The following essential development-tools are already ported:

Also the three addons that achieved the highest ranking during the latest poll at Ploneconf now work on Python 3:

You can also run the new frontend Volto with Plone on Python 3.

Open tasks

Before the final release of Plone 5.2 scheduled for February 2019 there are still a lot of things to do.

Here are the six most pressing issues:

  1. The upgrade- and porting-guide need to be completed.
  2. ZODB migration needs to be finished and documented.
  3. Much more addons need to be ported and released.
  4. We require performance tests.
  5. People who really need a replacement for FTP and/or WebDAV need to figure out how to do that.

There is still a lot to do. You can help Plone by coming to a sprint, testing it and fixing or reporting a bug.

Migrating to Python 3

From 1.1.2020 on you'll have to inform your clients that you are running their super-secure system on an unsupported version of Python. That might be ok if you're running a small website for a friend but for serious projects this simply will not do. Instead you should start to adopt Python 3 in your Plone projects now. You need to start early and plan ahead!

Here are the six steps you have to follow to upgrade to Python 3. Do not attempt to do all steps at once, instead you should work iteratively and even deploy to production in between steps.

1. Upgrade to Plone 5.2

You should still use Python 2.7 for this step. The changes between Plone 5.1 and Plone 5.2 are not huge but you will gain some great features including a new and much faster dropdown-navigation and a bootstrap-based ZMI.

The Upgrade-Guide for 5.2 still needs much more love. So if you encounter any issues please update that document and/or create a ticket in github.

2. Drop Archetypes and only use Dexterity

Archetypes is still supported on Plone 5.2 but only when running with Python 2.7. It is now officially deprecated and is not ported to Python 3. There has been a built-in migration for default types from Archetypes to Dexterity since 2013, and since 2015 there have been helpers and even a form that allows you to migrate your custom content.

3. Migrate your code to Python 3

Follow the documentation in to migrate your custom code and any addons that you need in your packages.

  • Make sure your code works in Python 2 and Python 3. If you are certain that you will not publish that code as a addon or reuse it in another project you can also drop support for Python 2. That will make porting and testing easier.
  • For most addons: Use python-modernize as described in the documentation.
  • For small addons you could even simply try to startup on Python 3 and and fix whatever fails (e.g. relative imports or invalid syntax)
  • Manually fix whatever python-modernize misses (e.g. relative imports)
  • Do the same with tests. You'll need a test-setup with tox and a test-matrix in travis that tests different Python versions. The setup in collective.ifttt is almost done and might be a good example.
  • If you have complex doctests consider migrating them to python tests since these are easier to get to pass in both Python 2 and 3. If you do want to keep them as doctests, change them so that the output for Python 3 is the default and use a Py23DocChecker to make the tests pass in Python 2 as well. See the example in plone.indexer.
  • Once all seems to work you need to test every single feature manually because even a high test coverage is never a guarantee that your code actually does what it should.

4. Migrate Addons and Dependencies

Now migrate any addons you use to Python 3 as explained above. The main difference is that now everyone who uses this addon benefits. You shoud not drop support for Python 2 for those!

I listed the addons that already run on Python 3 above. I'm sure that list will be outdated very soon. Before you start migrating an addon you need to check if there already is a branch that supports Python 3 and check if there is a ticket like the one for collective.easyform. If not create such a ticket and start the process.

5. Migrate your Database

ZODB itself is compatible with Python 3 but a database that was created in Python 2.7 cannot be used in Python 3 without being modified first. I will not go into details about that. You should read the documentation and see David Glick's talk.

The migration will probably require some downtime and in rare cases you might have to write your own mappings to tell zodbupdate how to handle your data.

6. Deploy on Python 3

Plone 5.2 will use WSGI with waitress as the default WSGI server. Alternatively you can use uwsgi or gunicorn. ZServer can only be used with Python 2.7.

Some deployment tools may not ported yet, others like ZRS or RelStorage already support Python 3. You should test your production setup early. There may be changes that you don't want to struggle with a week before a scheduled launch.

Summing up

Start that whole process as early as possible because you may need to deal with unexpected problems and addons that you did not know still used Archetypes. 2019 might be much busier for you than you expect.

Do not wait for a final release of Plone 5.2! Start the process of planning and testing a migration. Consinder hiring experts to help you if necessary.

Again: You can and you should work iteratively. That means you do not have to do everything in one step and one night:

  • Upgrade to 5.2 with Dexterity on Python 2 and deploy that early in 2019.
  • Then take some time to migrate and test your code and addons.
  • You can then deploy the Python 3-compatible code while still running Python 2.7.
  • Then you can migrate your existing database and deploy on Python 3.

FAQ

Here are some questions you might have:

What about Python 4?

That will not be a big issue since 4.0 will be a upgrade like from 3.7 to 3.8. The Python community simply don't want to release a Python 3.10.

When will Plone drop support for Python 2?

Not before we have a good reason. Maybe Plone 7? It would allow us to remove and clean up some code but we would not gain a lot.

What about Zope 5?

There are no real plans for that yet.

Will Archetypes be migrated to Python 3?

Please don't try to postpone the inevitable. We are too small a community to maintain two Python versions, two frontends and two content type frameworks.

How do we replace FTP/WebDAV?

I don't know, for WebDAV you might be able to use a middleware like WsgiDAV.

Why did you make all that effort and not 'simply' migrate to Guillotina?

Guillotina does not aim to be a replacement for Plone. Also it has no equivalent addon ecosystem, feature equality or migration path.

Can I now use async/await?

You can but it won't really help you since Plone and Zope are still fundamentally synchronous.

Why doesn't the Plone Foundation simply buy support for Python 2.7 from RedHat/IBM for all of us?

Are you kidding me?

Does Python 3 make Plone run faster?

Maybe. We still lack performance tests but my guess is that Plone 5.2 will run faster on Python 3 than on 2.

Can I run the same Database in Python 2 and 3?

No.You can neither use a DB create with py2 in py3 nor use one created with py3 in py2.

Is addon xyz already ported?

Most likely not.

Plone supports the latest two major version. Does that extend to Python?

Plone's security update policy supports the latest two major versions (currently, that means the 5.x series and the 4.3.x series) with security updates. That guarantee does not extend to the programming language, the operating systen or the hardware you are running Plone on.

Will this work on Windows?

Very likely. Someone will have to test it.

I need help with migrating our projects to Python 3. Who can help me?

Talk to us at Starzel.de or ask other professionals with experience.

Warm words

Migrating Plone to Python 3 was quite a ride and a great community effort. I had to cope with some brain-melting changes in Zope, fight heartbreaking test-isolation issues and toil through an endless list of packages and tests. But I had the immense privilege and pleasure to do all of that together with some of the smartest and nicest people I know. Thanks to all of you!

Watch the full talk if you want to learn more about how Plone finally landed on Python 3.

Here are all videos from Ploneconf 2018: Day 1, Day 2, Day 3.

Python 3, Plone 5.2, Zope 4 and more

erstellt von Philip Bauer zuletzt geändert: 2018-10-19T10:14:08+01:00
Report from the SaltLabs Sprint in Halle, Germany and the aftermath

With over 30 attendees the SaltLabs Sprint was among the bigger sprints in the history of Plone and Zope.

Fortunately, our host Gocept just moved into a new office and had plenty of space to spare. Stylish as we are, the Plone crowd quickly congregated in the palace-like salon with red walls and renaissance paintings. Even better: Throughout the week the cafe Kaffej downstairs was open for us to eat breakfast, lunch and for work. We tried to keep the barista busy producing excellent Cappuccino. Thanks!

Zope 4

The sprint split in two groups, one working on Zope and one working on Plone. A short report on about what happened at the Zope can be found at https://blog.gocept.com/2018/10/11/beta-permission-for-earl-zope-extended . Zope 4.0b6 was released just after the sprint. The new Beta 6 includes the new bootstrap-styled ZMI, support for Python 3.7 and a metric ton of bug fixes compared to 4.0b5 which was released in May this year. By the way: Zope can now be installed without buildout using pipenv

Python 3

The Plone crowd was huge and worked on several tasks. The biggest task - and the main reason to join the sprint - was continuing to port Plone to Python 3. Jens Klein, Peter Mathis, Alessandro Pisa, David Glick (remote) and Philip Bauer as lead worked mainly on fixing all remaining failing tests and getting the jenkins-builds to pass. We dove deep into crazy test-isolation-issues but also added a couple of missing features to Dexterity that were so far only tested with Archetypes.

During the sprint we got very close to a green build but a couple of really nasty test-isolation issues prevented a green build. A week after the sprint we finally got that done and all tests of Plone now pass in Python 2.7, Python 3.6 and Python 3.7 with the same code-base!

With the test passing we're now merging that work and preparing the release of Plone 5.2a1. The tasks for this are discussed in this ticket

Deprecate Archetypes

Archetypes will not be ported to Python 3 but still has to work when running Plone 5.2 in Python 2.7. Since all tests that used PloneTestCase were still using Archetypes Philip changed that to use Dexterity now and created a new testlayer (plone.app.testing.bbb_at.PloneTestCase) that is now used by all the packages using the Archetypes-stack. See https://github.com/plone/plone.app.testing/pull/51. You could use this layer in your archetypes-based addon-packages that you want to port to Plone 5.2.

Addons

Yes, Plone is now able to run on Python 3. But without the huge ecosystem of addons that would be of limited real-world use. Jan Mevissen, Franco Pellegrini (remote) and Philip Bauer ported a couple of addons to Python 3 that Plone developers use all the time:

Being able to run tests against Plone in Python 3, and with these development-tools now available also, it is now viable to start porting other addons to Python 3. To find out how hard that is I started porting collective.easyform. The work is only 97% finished since some tests need more work but it is now useable in Python 3!

ZODB Migration

Harald Friesenegger and Robert Buchholz worked on defining a default migration-story for existing databases using zodbupdate. To discuss the details of this approach we had a hangout with Jim Fulton, David Glick and Sylvain Viollon. They solved a couple of tricky issues and wrote enough documentation. This approach seems to work well enough and the documentation points out some caveats but the ZODB migration will require some more work.

Relevant links:

ZODB Issues

Sune Broendum Woeller worked on a very nasty and complex issue with KeyErrors raising while releasing resources of a connection when closing the ZODB. That happened pretty frequently in our test-suites but was so far unexplained. He analyzed the code and finally was able to add a failing test to prove his theory. Then Jim Fulton realized the problem and wrote a fix for it. This will allow us to update to the newest ZODB-version once it is released. See https://github.com/zopefoundation/ZODB/issues/208 for details.

WSGI-Setup

Thomas Schorr added the zconsole module to Zope for running scripts and an interactive mode. Using WSGI works in Python 2 and Python 3 and will replace ZServer in Python 3.

Frontend and Theming

Thomas Massman, Fred van Dijk, Johannes Raggam and Maik Derstappen looked into various front-end issues, mainly with the Barceloneta theme. They closed some obsolete tickets and fixed a couple of bugs.

They also fixed some structural issues within the Barceloneta theme. The generated HTML markup now has the correct order of the content columns (main, then left portlets, then right portlets) which allows better styling for mobile devices. Also in the footer area we are now able to add more portlets to generate a nice looking doormat. See the ticket https://github.com/plone/plonetheme.barceloneta/pull/163 for screenshots and details.

They also discussed a possible enhancement of the Diazo based theming experience by including some functionality of spirit.plone.theming into plone.app.theming and cleaning up the Theming Control Panel. A PLIP will follow for that.

New navigation with dropdown support

Peter Holzer (remote) continued to work on the new nativation with dropdown-support which was started on collective.navigation by Johannes Raggam. Due to the new exclude_from_nav index and optimized data structures the new navigation is also much faster than other tree based navigation solutions (10x-30x faster based on some quick tests).

Static Resource Refactoring

Johannes Raggam finished work on his PLIP to restructure static resources. With this we no longer carry 60MB of static resources in the CMFPlone repository, it allows to use different versions of mockup in Plone, enable us to release our libraries on npm and will make it easier to switch to a different framework. Skin scripts removal Katja Süss, Thomas Lotze, Maurits van Rees and Manuel Reinhardt worked hard at removing the remaining python_scripts. The work is still ongoing and it would be great to get rid of the last of these before a final Plone 5.2 release. See https://github.com/plone/Products.CMFPlone/issues/1801 for details. Katja also worked on finally removing the old resource registry (for js and css).

Test parallelization

Joni Orponen made a lot of progress on speeding up our test-runs by running different test-layers in parallel. The plan is to get them from 30-60 minutes (depending on server and test-setup) to less than 10 minutes. For a regularly updated status of this work see https://community.plone.org/t/ci-run-speedups-for-buildout-coredev/6225

Documentation and User-Testing

Paul Roeland fought with robot-tests creating the screenshots for our documentation. And won. A upgrade guide to Plone 5.2 and Python 3 was started by several people. Jörg Zell did some user-testing of Plone on Python 3 and documented some errors that need to be triaged.

Translations

Jörg Zell worked on the german translations for Plone and nearly got to 100%. After the sprint Katja Süß did an overall review of the german translation and found some wording issues with need for a discussion

Plone React

Rob Gietema and Roel Bruggink mostly worked on their trainings for React and Plone-react, now renamed to “Volto”. Both will be giving these trainings at Plone Conference in Tokyo. On the second day of the sprint Rob demoed the current state of the new react-based frontend for Plone.

Assorted highlights

  • In a commit from 2016 an invisible whitespace was added to the doctests of plone.api. That now broke our test in very obscure ways. Alessandro used some dark magic to search and destroy.
  • The __repr__ for persistent objects changed breaking a lot of doctests. We still have to figure out how to deal with that. See https://github.com/zopefoundation/zope.site/issues/8 for details.
  • There was an elaborate setup to control the port during robot-tests. By not setting a port at all the OS actually takes care of this to makes sure the ports do not conflict. See https://github.com/plone/plone.app.robotframework/pull/86. Less is sometimes more.
  • Better late than never: We now have a method safe_nativestring in Products.CMFPlone.utils besides our all-time favorites safe_unicode and safe_encode. It transforms to string (which is bytes in Python 2 and text in Python 3). By the way: There is also zope.schema.NativeString and zope.schema.NativeStringLine.
  • We celebrated the 17th birthday of Plone with a barbecue and a generous helping of drinks.

Porting Plone to Python 3

erstellt von Philip Bauer zuletzt geändert: 2018-05-24T09:40:11+01:00
&tldr; Plone is running nicely on Python 3 but now we have to fix over 500 failing tests.

Since I wrote the proposal to Port Plone to Python 3, so much has happened that a status update is needed.

Let's step back a little: The first steps towards Python 3 were taken during the sprint at the Plone conference in Barcelona. The epic PLIP to update to Zope 4 was merged and we started porting individual packages to Python 3 without being able to run any tests. With the help of sixer and python-modernize we tried to get the most obvious import and syntax issues out of the way. That work was continued at the Alpine City Sprint in Innsbruck and in the end, we treated more than 150 packages like this. Some of this work is documented in the Plone tracker.

Along the way I wrote a best-practice guide on how to port Plone packages this way.

As the PLIP states, there are several approaches to porting Plone to Python 3:

  1. Migrate and test packages that have no dependency to CMFPlone and test them in Python 2 and 3.
  2. Prepare packages for Python 3 without being able to test them in Python 3.
  3. Start up Plone on Python 3 and fix whatever breaks. When start-up works, create a Plone Site and again, fix whatever breaks.
  4. Port plone.testing and plone.app.testing to Python 3 and start running tests. Fix what breaks during the setup of the layers.
  5. Run the tests with Python 3 and fix all broken tests.

At the sprint in Innbruck I started with #3 and kept going after the sprint until I was able to create an instance. At the Plone Tagung in Berlin I was able to demo the creation of a site but nothing was rendered yet.

After that, I kept going and I was finally able to create a site and manage content, which is what a CMS is about. It looked a bit raw but I was able to add and edit some content - yay!

early screenshot of plone with python 3

Work continued at an unsteady pace, and with important contributions from Michael Howitz, Alessandro Pisa and David Glick, things started to get better. Even the theme and js started working. Slowly broken features became the exception, not the rule.

Last week at the Zope 4 Welcome Sprint in Halle we removed all feature blockers that we had found so far. Plone with Python 3 looks and feels like it used to in Python 2. During the sprint there was also a lot of progress on:

  • the wsgi setup
  • logging and tracebacks when using wsgi
  • porting plone.testing
  • a new theme for the ZMI
  • beta releases for many packages.

There was also some progress on the difficult issue of database migrations. It seems like zodbupdate is the best tool to do that but there is probably a lot of work ahead.

Read more about the sprints outcome in the blogpost by Michael Howitz.

There is a Jenkins job and as of today, it is running (not passing) tests for all packages (except Archetypes) with Python 3.

At the moment we run 6594 tests from 115 packages with 257 failures and 315 errors - not bad when you keep in mind that we still focus on features, not on tests. Tests for a couple of packages are green, including plone.api which is great since plone.api uses most of our stack one way or another. Jenkins runs every three hours and here you can see the progress over time.

Here is a screenshot from today that shows Plone doing some pretty non-trivial things: Editing a recurring Event and also Richtext with Images and Links:

screenshot today

Next steps:

  • Create a demo site running on Python 3 that people can use to find broken features. This will happen at the Plonator Sprint in Munich.
  • Fix all tests that look like they fail because of a broken feature.
  • Fix all remaining tests to pass in Python 2 and Python 3. Since there are a gazillion doctests that will take some time.
  • Port plone.app.robotframework to python 3 and fix robottests.
  • Experiment with porting a ZODB with Plone to Python 3.

If you want to participate you can simply look at the failing tests and start fixing them. You can also try to fix one of the open issues.
The setup of a coredev environment with Python 3 is really simple and documented.