Blog

Hello Thomas!

erstellt von Philip Bauer zuletzt geändert: 2017-03-01T17:18:25+01:00
We're excited to welcome Thomas Lotze to the team of Starzel.de.

Thomas has many years of experience developing complex web projects with Python, especially using Pyramid and Plone. His interests also include automated testing and questions of software infrastructure. He shares our dedication to Open Source and we are looking forward to many exciting projects in the future.

Aside from development, Thomas does trainings on subjects that he knows well from day-to-day work, in particular Python, Pyramid and pytest.

You can meet the whole team at World Plone Day and PyConWEB in Munich.

World Plone Day - 28. April 2017 - München

erstellt von Steffen Lindner zuletzt geändert: 2017-02-17T21:12:59+01:00
Der World Plone Day ist eine weltweite Initiative der großartigen Community des Open Source Content-Management-Systems Plone.

Merkt euch den 28. April 2017 in euren Kalendern vor, an diesem Tag findet der World Plone Day in München statt. Die Location steht schon fest: Uni München, Oettingenstraße, Raum L 155.

Wir werden uns den halben Tag (14 bis 18 Uhr) intensiv mit Plone beschäftigen. Jeder der sich für CMS, Plone, Python und das Web interessiert, ist eingeladen. Die Teilnahme ist kostenlos. Am Programm feilen wir noch, möglich sind diese Themen: Plone 5.1, Quaive, PLOG und Shopsysteme mit Plone.

Details zum Programm sind hier zu finden.

Bei schönem Wetter geht es danach in den Biergarten am Chinesischer Turm!

Eine Anmeldung schalten wir bis Ende Feburar auf plone.de online.

Stay tuned!

Towards Plone 6

erstellt von Philip Bauer zuletzt geändert: 2017-03-01T11:52:13+01:00
A report from the Alpine City Sprint in Innsbruck

Two of the main items on the Plone roadmap are the move to Python 3 and a more active role in the development of the underlying stack of Plone, namely Zope. After the recent uptake of activity in development of Zope (see here and here) it was a sensible first task to try to get Plone to run on the most recent Zope stack. What exactly does that mean?

There is a PLIP that proposes to upgrade Plone's Zope-related dependencies and is approval by the Plone Framework Team to be a part of Plone 6. As with all PLIPs, the point is not only to agree on a task and a implementation but also actually doing the work. After two smaller efforts at sprints last year where only 2 or 3 people were working on the task we finally got a critical mass of developers together for the Alpine City Sprint. 15 people sprinted and worked on this one task.

Off to a good start

On the day before the sprint I rebased some of our branches and with a couple of quick and really dirty fixes I got Plone run on Zope 4. Many features were still not working but a working instance certainly provided motivation for this very dedicated sprint.

WSGI replaces the ZServer

The ZServer is now a optional dependency and the new WSGI-Publisher is the default publisher making Plone fully WSGI-compliant! Thomas Schorr added some necessary buildout-parts to run Plone on uwsgi, waitress and gunicorn. Documentation is here and examples are here. Even the testbrowser used for all of Plone's functional tests now uses the WSGI-Publisher. The advantages of WSGI are obvious: It makes it easier to have interact Plone with other services and some features can be developed as WSGI-middleware. It'll be interesting to see if we will make more use of WSGI for Plone in the future than "simply" for serving.

ZCatalog 4 and ZODB 5

The result of the sprint is beyond what was originally proposed in the PLIP. We now not only use the master branch from Zope and the zope-toolkit packages (zope.interface, zope.component etc.) but also ZODB 5 and ZCatalog 4.

One change that required some work was that ZCatalog, as it now returns a empty list when you pass no query or a invalid query. Especially in the cases when you query the catalog for a non-existing index (portal_typos='foo') getting all brains from the Catalog seems like a bad idea. The down-side that change there is that there is no obvious way to walk through all brains in the catalog any more. Johannes Raggam added the convenience-method catalog_get_all for this.

The changes in ZODB are numerous but for Plone-developers nothing much really changes except that is supposed to be faster and supports Python >= 3.3 and 2.7. For the future it will be also interesting to discuss if switching to NewtDB would be a option. NewtDB runs ZODB 5 with RelStorage and stores data as json in Postgres.

Python 3

The sprint did not try to move Plone to Python 3 yet but rather clear the way so we can do it. The last remaining blocker with C-code was RestrictedPython and Alexander Loechel and Michael Howitz continued working on the rewrite based on the Python AST module. At the end of the sprint they were happy to announce that it is no longer a blocker for our move to Python 3. All the other packages that rely on C-code that seems mysterious to mere Python-mortals (ExtensionClass, AccessControl, Acquisition, Persistence) were already ported to Python 3. RestrictedPython is a very useful tool that allows you to execute untrusted Python code that is loaded within a running Python program. It checks and modifies source code to only allow the execution of a restricted subset of Python.

What remains to be done?

  • Fix some failing tests (DONE): As of today we are down to 8 failing tests 6 failing tests 2 failing tests 0 failing tests!!! 
  • Update tasks and tickets
  • Merge changes: A lot of the changes were already merged in the master-branches for the current coredev (for Plone 5.1). But some merges will have to wait until Plone 5.1 is released because they only target Plone 6.0. Once all test are green and all our zope-related branches are merged we can make a release of Zope 4.0a3 and switch from using source-checkouts for everything (auto-checkout = *) to using released versions.
  • Documentation: We need to document what changed in Zope from the perspective of Plone. Most notably some lesser features of the ZMI are gone. These and other changes need to go into the upgrade-guide for the next Plone-Version. Also the new setup for various WSGI-Servers needs to be well-documented so that developers can  copy&paste best-practice configs into their buildouts. The documentation was started in a Google Doc where everyone can contribute. Once it's stable we'll move it to their respective places.
  • Upgrades: So far it seems like the changes we made did not make any upgrade-steps necessary. After adding a quick hack to work around the missing zope-controlpanel I was able to upgrade a existing Plone 5.0.6 site without any issues.
  • Profiling: Initial results with from plone.instanceprofiling indicate the speed of Plone has increased but we need more real-life tests before a final judgement. It will also be interesting to see the impact of Python 3 on speed.

Inside the sprint

As usual during the sprint we ran into a bunch of blockers and hard nuts to crack. The hardest one was probably that the current users roles were mysteriously dropped during a request after running trough some C-code. Like the memory of a bad dream, the bug vanished in the light of the morning - actually someone switched to different branch of AccessControl. On the first day already, Matthew Wilkes had committed to a metaclass in Persistence (thankfully brown-bagging my terrible late-night hack and replacing it with something much more elegant), and installed gdb to debug the issue with the dropped roles.

Unfortunately many of our tests were broken since the testlayer did not use chameleon, thankfully after a lot of debugging the solution found by Thomas Schorr was pretty simple. Also tough was a issue that broke tests in weird ways. Gladly Jens Klein, our host, dug deep and found the culprit: Combining bundles modified the response's Content-Type header, surprise! A issue similar to https://github.com/plone/Products.CMFPlone/pull/1438.

Shops!

By the way: We also had a demo of bda.plone.shop and a discussion about shop-solutions for Plone. bda.plone.shop has plenty of great features and seems really well thought-through. It still lacks some end-user documentation with screenshots and a new release but we will certainly use it in our next project that involves a shop.

Where do we go from here?

To the pool in Sorrento! Plone Open Garden (PLOG) in beautiful Sorrento is the perfect opportunity to discussing the further roadmap to Python 3 and continue working on this. PLOG will have a strong focus on the headless CMS project for which a Python 3-compatible Plone 6 will be a mayor milestone. Register now and enjoy a week at the pool with the Plone community.

And then on to Finnland to the Midsummer Sprint! All the work on the backend needs to be balanced with some fun UX and UI work to enhance the content editors experience.

Summary

We worked on many important parts of the Plone Roadmap: Removing unused components, clean up the code base, moving to Python 3, and making Plone WSGI compliant. There is now a good chance that Plone will be on Python 3 this or next year. A great sprint, dedicated and smart people, good beer and food and nice mountains. Thanks to the organizers of Klein und Partner who worked hard to keep us fed, fueled and entertained and thanks to all the sprinters who dedicated so much of their time for such a great open source project!

Update (19.02.2017)

  1. Last week gocept announced there will be another Zope sprint in Mai with the goal to port Zope to Python 3. Already 11 developers signed up.
  2. David Glick fixed the remaining failing tests (mosty issues with the new handling of exceptions).

 

Push coverage report out of an Gitlab CI Runner

erstellt von Steffen Lindner zuletzt geändert: 2017-02-06T17:26:17+01:00
For our Plone/Python projects we often generate coverage reports as HTML sites, this posts show how you can push this report out of the Gitlab CI Runner.

 Save FTP Password & Login as secret variables in Gitlab > Settings > Variables. They can be accessed in the .gitlab-ci.yml as $FTPLOGIN & $FTPPASSWORD.

 

Here is our .coveragerc file, which sets the output directory:

 [report]
include =
    src/plone.site/*
omit =
    */test*
    */upgrades/*
    */browser/dummy.py
[html]
directory = parts/test/htmlreport
title = Plone Project

And our .gitlab-ci.yml:

before_script:
    - /usr/bin/virtualenv .
    - ./bootstrap_ci.sh
job1:
  script:
     - export DISPLAY=:99
     - ./bin/code-analysis
     - ./bin/coverage erase
     - ./bin/coverage run -p --source=src bin/test || exit 1
     - ./bin/coverage combine
     - ./bin/coverage html
     - ./bin/coverage report
        # Stage robot
     - ./bin/test --all -t 'robot' || exit 1
     - cd parts/test/htmlreport/ && find . -type f -exec curl --ftp-create-dirs -T {} -u $FTPLOGIN:$FTPPASSWORD ftp://45.54.45.54/uhlmann/$CI_BUILD_ID/{} \;
  artifacts:
        expire_in: 1 week
        paths:
            - parts/test/

The interesting part is the curl one, it uploads all coverage files to a FTP server.

If your static webserver is public accessible use basic auth with a .htaccess file:

htpasswd -c .htpasswd starzel

.htaccess file:

AuthUserFile /var/www/htdocs/.htpasswd
AuthGroupFile /dev/null
AuthName "Please Enter Password"
AuthType Basic
Require valid-user

Better integration into Gitlab coming soon as Gitlab Pages see this Pull Request.

Todo:

  - Cron to delete old directories.

  - Upload files if robot / code-analysis failed.

 

Fix failing parallel running browser tests with Gitlab & Robotframework

erstellt von Steffen Lindner zuletzt geändert: 2017-01-25T16:41:19+01:00
One of our project is organized in sprints where all developer work on the same code at the same time. We use one Gitlab CI server with a simple shell executer and had random failing builds with Robotframwork & Firefox.

Our Gitlab CI Runner can run three shell executer at the same time. Sometimes two Robotframework (Selenium) tests run in parallel and one of them fails with this error:

error: [Errno 98] Address already in use

That means Firefox is already in use (by the other build) and the CI job failed and blocked our process. Gitlabs automerge -feature does not happen and the code-reviewer needs to manually start the job again since it is marked as a fail.

We solved the problem with a little shell script isfirefoxinuse.sh that waits if a Robot process is running:

#!/bin/sh

while :
do

RESULT=`pgrep -f robot`

if [ "${RESULT:-null}" = null ]; then
echo "Robot not running, starting "
./bin/test --all -t robot || exit 1
break
else
echo "running"
fi
sleep 20
done

Our .gitlab-ci.yml file looks like this:

before_script:
- export PATH=$PATH:bin/
- export DISPLAY=:99

build_project:
# stage: build
script:
# Stage: build
- /usr/bin/virtualenv .
- ./bootstrap_ci.sh
# Stage test
- ./bin/code-analysis
- ./bin/i18nize_all && ./bin/podiff_all
- ./bin/coverage erase
- ./bin/coverage run -p --source=src bin/test || exit 1
- ./bin/coverage combine
- ./bin/coverage html
- ./bin/coverage report
# Stage robot
- ./bin/isfirefoxinuse.sh || exit 1
artifacts:
expire_in: 1 week
paths:
- parts/test/

The call to start the robot tests (./bin/test --all -t robot) is wrapped into the ./bin/isfirefoxinuse.sh script.

If you are a buildout user, this might be useful to get the script integrated:

parts +=
isfirefoxinuse
...

recipe = collective.recipe.cmd
on_install = true
on_update = true
cmds = cp ${buildout:directory}/templates/isfirefoxinuse.in ${buildout:bin-directory}/isfirefoxinuse.sh

 Hope this helps someone!