About life and work after the digital revolution

Plone Connection Podcast: Episode 01 - Philip Bauer

erstellt von Philip Bauer zuletzt geändert: 2020-11-19T17:56:39+01:00
I was honoured to have been invited as the guest on the inaugural episode of the Plone Connection Podcast!

The Plone Connection Podcast is a monthly podcast produced by Six Feet Up. Every month, Six Feet Up's Director of Engineering T. Kim Nguyen sits down with a different member of the Plone Community and asks them about their work with or on the Plone CMS.

Many thank to my good friend Kim for doing this!

Your first Plone 6 Project

erstellt von Philip Bauer zuletzt geändert: 2020-11-18T14:25:39+01:00
In this talk I picked some tasks from the Mastering Plone Training and showed how you can meet the same requirements using Volto, the new React-based front end for Plone.

I've had the opportunity to give this talk at the most excellent Python Web Conference.

ZODB Database debugging

erstellt von Philip Bauer zuletzt geändert: 2020-08-25T08:25:32+01:00
Debugging issues in the ZODB is one of the hardest tasks and can drive developers to insanity. Especially after the migration of projects to newer Versions of Plone or Python these skills are needed. Here is most of what I learned about this topic.

The problem

The ZODB contains python objects serializes as pickles. When a object is loaded/used a pickle is deserialized ("unpickled") into a python object.

A ZODB can contain objects that cannot be loaded. Reasons for that may be:

  • Code could not be loaded that is required to unpickle the object (e.g. removed packages, modules or classes)
  • Objects are referenced but missing from the database (e.g. a blob is missing)
  • The objects contains invalid entries (e.g. a reference to a oid that uses a no longer supported format)

The most frequent issues are caused by:

  • Improperly uninstalled or removed packages (e.g. Archetypes, ATContentTypes, CMFDefault, PloneFormGen etc.)
  • Code has changed but not all objects that rely on that code as updated
  • Code was refactored and old imports are no longer working

You should not blame the migration to Python 3 for these issues! Many issues may already exists in their database before the migration but people usually do not run checks to find issues. After a migration to Python 3 most people check their database for the first time. This may be because the documentation on python3-migration recommends running the tool zodbverify.

Real problems may be revealed at that point, e.g when:

  • Packing the Database fails
  • Features fail

You can check you ZODB for problems using the package zodbverify. To solve each of the issues you need to be able to answer three questions:

  1. Which object is broken and what is the error?
  2. Where is the object and what uses it?
  3. How do I fix it?

In short these approaches to fixing exist:

  1. Ignore the errors
  2. Add zodbupgrade mappings
  3. Patch your python-path to work around the errors
  4. Replace broken objects with dummies
  5. Remove broken objects the hard way
  6. Find our what and where broken objects are and then fix or remove them safely

I will mostly focus on the last approach.

But before you spend a lot of time to investigate individual errors it would be a good idea to deal with the most frequent problems, especially IntIds and Relations (see the chapter "Frequent Culprits") below. In my experience these usually solved most issues.

Find out what is broken

Check your entire database

Use zodbverify to verify a ZODB by iterating and loading all records. zodbverify is available as a standalone script and as addon for plone.recipe.zope2instance. Use the newest version!

In the simplest form run it like this:

$ bin/zodbverify -f var/filestorage/Data.fs

It will return:

  • a list of types of errors
  • the number of occurences
  • all oids that raise that error on loading


zodbverify is only available for Plone 5.2 and later. For older Plone-Versions use the scripts and from the ZODB package:

$ ./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs
$ ./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs

The output of zodbverify might look like this abbreviated example from a medium-sized intranet (1GB Data.fs, 5GB blobstorage) that started with Plone 4 on Archetypes and was migrated to Plone 5.2 on Python 3 and Dexterity:

$ ./bin/zodbverify -f var/filestorage/Data.fs


INFO:zodbverify:Done! Scanned 163955 records.
Found 1886 records that could not be loaded.
Exceptions, how often they happened and which oids are affected:

ModuleNotFoundError: No module named 'Products.Archetypes': 1487
0x0e00eb 0x0e00ee 0x0e00ef 0x0e00f0 0x0e00f1 0x2b194b 0x2b194e 0x2b194f 0x2b1950 [...]

ModuleNotFoundError: No module named 'Products.PloneFormGen': 289
0x2b1940 0x2b1941 0x2b1942 0x2b1943 0x2b1944 0x2b1974 0x2b1975 0x2b1976 0x2b1977 [...]

AttributeError: module 'App.interfaces' has no attribute 'IPersistentExtra': 34
0x2c0a69 0x2c0a6b 0x2c0ab7 0x2c0ab9 0x2c555d [...] 0x35907f

ModuleNotFoundError: No module named 'Products.CMFDefault': 20
0x011e 0x011f 0x0120 0x0121 0x0122 0x0123 0x0124 0x0125 0x0126 0x0127 0x0128 0x0129 0x012a 0x012b 0x012c 0x012d 0x012e 0x012f 0x0130 0x0131

ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 20
0x3b1cde 0x3b1ce0 0x3b1ce4 0x3b1ce6 0x3b1ce9 0x3b1ceb 0x3b1cee 0x3b1cf0 0x3b1cf4 0x3b1cf6 0x3b1cf9 0x3b1cfb 0x3b1cfe 0x3b1d00 0x3b1d04 0x3b1d06 0x3b1d09 0x3b1d0b 0x3b1d0e 0x3b1d10

ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16
0x2c0a68 0x2c0a6a 0x2c555c 0x2c555e 0x2c560b 0x2c560d 0x2c5663 0x2c5665 0x2c571b 0x2c571d 0x2c5774 0x2c5776 0x2c5833 0x2c5835 0x33272d 0x33272f

ModuleNotFoundError: No module named 'fourdigits': 8
0x28030f 0x280310 0x280311 0x280312 0x280313 0x280314 0x280315 0x280316

ModuleNotFoundError: No module named 'Products.ATContentTypes': 4
0x0e00e9 0x0e011a 0x0e01b3 0x0e0cb3

AttributeError: module '' has no attribute 'IEventSettings': 3
0x2a712b 0x2a712c 0x2a712d

ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

ModuleNotFoundError: No module named '': 1

ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

You can see all different types of errors that appear and which objects are causing them. Objects are referenced by their oid in the ZODB. See the Appendix on how to deal with oids.

You can see that among other issues there are still a lot of references to Archetypes and PloneFormGen (I omitted the complete lists) even though both are no longer used in the site.

Before the summary the log dumps a huge list of errors that contain the pickle and the error:

Could not process unknown record 0x376b77 (b'\x00\x00\x00\x00\x007kw'):
INFO:zodbverify:b'\x80\x03cProducts.PloneFormGen.content.thanksPage\nFormThanksPage\nq\x00.\x80\x03}q\x01(X\x0c\x00\x00\x00showinsearchq\x02\x88X\n\x00\x00\x00_signatureq\x03C\x10\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7q\x04X\r\x00\x00\x00creation_dateq\x05cDateTime.DateTime\nDateTime\nq\x06)\x81q\x07GA\xd6\xdf_\xba\xd56"\x89X\x05\x00\x00\x00GMT+2q\x08\x87q\tbX\r\x00\x00\x00marshall_hookq\nNX\n\x00\x00\x00showFieldsq\x0b]q\x0cX\x02\x00\x00\x00idq\rX\t\x00\x00\x00thank-youq\x0eX\x11\x00\x00\x00_at_creation_flagq\x0f\x88X\x11\x00\x00\x00modification_dateq\x10h\x06)\x81q\x11GA\xd6\xdf_\xba\xd7\x15r\x89h\x08\x87q\x12bX\x05\x00\x00\x00titleq\x13X\x05\x00\x00\x00Dankeq\x14X\x0f\x00\x00\x00demarshall_hookq\x15NX\x0e\x00\x00\x00includeEmptiesq\x16\x88X\x0e\x00\x00\x00thanksEpilogueq\x17C\x08\x00\x00\x00\x00\x007k\xaaq\x18cProducts.Archetypes.BaseUnit\nBaseUnit\nq\x19\x86q\x1aQX\x07\x00\x00\x00showAllq\x1b\x88X\x12\x00\x00\x00_EtagSupport__etagq\x1cX\r\x00\x00\x00ts34951147.36q\x1dX\x0b\x00\x00\x00portal_typeq\x1eX\x0e\x00\x00\x00FormThanksPageq\x1fX\x0b\x00\x00\x00searchwordsq C\x08\x00\x00\x00\x00\x007k\xabq!h\x19\x86q"QX\x07\x00\x00\x00_at_uidq#X \x00\x00\x00a2d15a36a521471daf2b7005ff9dbc62q$X\r\x00\x00\x00at_referencesq%C\x08\x00\x00\x00\x00\x007k\xacq&cOFS.Folder\nFolder\nq\'\x86q(QX\x0e\x00\x00\x00thanksPrologueq)C\x08\x00\x00\x00\x00\x007k\xadq*h\x19\x86q+QX\x0f\x00\x00\x00noSubmitMessageq,C\x08\x00\x00\x00\x00\x007k\xaeq-h\x19\x86q.QX\x03\x00\x00\x00_mdq/C\x08\x00\x00\x00\x00\x007k\xafq0cPersistence.mapping\nPersistentMapping\nq1\x86q2QX\x12\x00\x00\x00__ac_local_roles__q3}q4X\x16\x00\x00\x00xxx@xxx.deq5]q6X\x05\x00\x00\x00Ownerq7asu.'
INFO:zodbverify:Traceback (most recent call last):
  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/", line 62, in verify_record
    class_info = unpickler.load()
  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/", line 62, in find_class
    return super(Unpickler, self).find_class(modulename, name)
ModuleNotFoundError: No module named 'Products.PloneFormGen'

Inspecting a single object

In this case the object with the oid 0x376b77 seems to be a FormThanksPage from Products.PloneFormGen. But wait! You deleted all of these, so where in the site is it?

If the offending object is normal content the solution is mostly simple. You can call obj.getPhysicalPath() to find out where it is. But ore often than not editing and saving will fix the problem. In other cases you might need to copy the content to a new item and delete the broken object.

But usually it is not simply content but something else. Here are some examples:

  • A annotation on a object or the portal
  • A relationvalue in the relatopn-catalog
  • A item in the IntId catalog
  • A old revision of content in CMFEditions
  • A configuration-entry in portal_properties or in portal_registry

The hardest part is to find out what and where the broken object actually is before removing or fixing it.

The reason for that is that a entry in the ZODB does not know about it's parent. Acquisition finds parents with obj.aq_parent() but many items are not-Acquisition-aware. Only the parents that reference objects know about them.

A object x could be the attribute some_object on object y but you will not see that by inspecting x. Only y knows that x is y.some_object.

A way to work around this is used by the script on ZODB. It allows you to list all incoming and outgoing references to a certain object.

With this you will see that x is referenced by y. With this information you can then inspect the object y and hopefully see how x is set on y.

More often than not y is again not a object in the content-hierarchy but maybe a BTree of sorts, a pattern that is frequently used for effective storage of many items. Then you need to find out the parent of y to be able to fix x.

And so forth. It can a couple of steps until you end up in a item that can be identified, e.g. portal_properties or RelationCatalog and usually only exists once in a database.

To make the process of finding this path less tedious I extended zodbverify in with a feature that will show you all parents and their parents in a way that allows you to see where in the tree is it.

Before we look at the path of 0x376b77 we'll inspect the object.

Pass the oid and the debug-flag -D to zodbverify with ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D:

$ ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D

INFO:zodbverify:Inspecting 0x376b77:
<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>
Object as dict:
{'__Broken_newargs__': (), '__Broken_state__': {'showinsearch': True, '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7', 'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'), 'marshall_hook': None, 'showFields': [], 'id': 'thank-you', '_at_creation_flag': True, 'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'), 'title': 'Danke', 'demarshall_hook': None, 'includeEmpties': True, 'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>, 'showAll': True, '_EtagSupport__etag': 'ts34951147.36', 'portal_type': 'FormThanksPage', 'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>, '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62', 'at_references': <Folder at at_references>, 'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>, 'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>, '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>, '__ac_local_roles__': {'': ['Owner']}}}
The object is 'obj'
[2] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/
-> pickle, state = storage.load(oid)

Even before you use the provided pdb to inspect it you can see that it is of the class persistent broken, a way of the ZODB to give you access to objects even though their class can no longer be imported.

You can now inspect it:

(Pdb++) obj
<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>
(Pdb++) pp obj.__dict__
{'__Broken_newargs__': (),
 '__Broken_state__': {'_EtagSupport__etag': 'ts34951147.36',
                      '__ac_local_roles__': {'': ['Owner']},
                      '_at_creation_flag': True,
                      '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62',
                      '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>,
                      '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7',
                      'at_references': <Folder at at_references>,
                      'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'),
                      'demarshall_hook': None,
                      'id': 'thank-you',
                      'includeEmpties': True,
                      'marshall_hook': None,
                      'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'),
                      'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>,
                      'portal_type': 'FormThanksPage',
                      'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>,
                      'showAll': True,
                      'showFields': [],
                      'showinsearch': True,
                      'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>,
                      'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>,
                      'title': 'Danke'}}

If you now choose to continue (by pressing c) zodbverify it will try to disassemble the pickle. That is very useful for in-depth debugging but out of the scope of this documentation.

Inspect the path of references

Now you know it is broken but you still don't know where this ominous FormThanksPage actually is.

Continue to let zodbverify find the path to the object:

INFO:zodbverify:Building a reference-tree of ZODB...
INFO:zodbverify:Created a reference-dict for 163955 objects.

This oid is referenced by:

INFO:zodbverify:0x376ada BTrees.IOBTree.IOBucket at level 1
INFO:zodbverify:0x28018c BTrees.IOBTree.IOBTree at level 2
INFO:zodbverify:0x280184 five.intid.intid.IntIds at level 3
INFO:zodbverify:0x1e five.localsitemanager.registry.PersistentComponents at level 4
INFO:zodbverify:0x11 Products.CMFPlone.Portal.PloneSite at level 5
INFO:zodbverify:0x01 OFS.Application.Application at level 6
INFO:zodbverify: 8< --------------- >8 Stop at root objects

INFO:zodbverify:0x02f6 persistent.mapping.PersistentMapping at level 7
INFO:zodbverify: 8< --------------- >8 Stop at root objects

INFO:zodbverify:0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at level 8
INFO:zodbverify: 8< --------------- >8 Stop at root objects

INFO:zodbverify:0x02f5 at level 6
INFO:zodbverify:0x02fa zope.ramcache.ram.RAMCache at level 7
INFO:zodbverify:0x02fd at level 8
INFO:zodbverify:0x338f13 at level 9
INFO:zodbverify:0x0303 BTrees.OOBTree.OOBTree at level 10
INFO:zodbverify:0x346961 at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify:0x02fe at level 9
INFO:zodbverify:0x034d plone.keyring.keyring.Keyring at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify:0x376864 BTrees.IOBTree.IOBucket at level 3
INFO:zodbverify:0x31049f BTrees.IOBTree.IOBucket at level 4
INFO:zodbverify:0x325823 BTrees.IOBTree.IOBucket at level 5
INFO:zodbverify:0x3984c8 BTrees.IOBTree.IOBucket at level 6
INFO:zodbverify:0x2cce9a BTrees.IOBTree.IOBucket at level 7
INFO:zodbverify:0x2c6669 BTrees.IOBTree.IOBucket at level 8
INFO:zodbverify:0x2c62b4 BTrees.IOBTree.IOBucket at level 9
INFO:zodbverify:0x2c44c1 BTrees.IOBTree.IOBucket at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify:0x377536 BTrees.OIBTree.OIBucket at level 2
INFO:zodbverify:0x376b14 BTrees.OIBTree.OIBucket at level 3
INFO:zodbverify:0x376916 BTrees.OIBTree.OIBucket at level 4
INFO:zodbverify:0x376202 BTrees.OIBTree.OIBucket at level 5
INFO:zodbverify:0x373fa7 BTrees.OIBTree.OIBucket at level 6
INFO:zodbverify:0x37363a BTrees.OIBTree.OIBucket at level 7
INFO:zodbverify:0x372f26 BTrees.OIBTree.OIBucket at level 8
INFO:zodbverify:0x372cc8 BTrees.OIBTree.OIBucket at level 9
INFO:zodbverify:0x36eb86 BTrees.OIBTree.OIBucket at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify:0x407185 BTrees.OIBTree.OIBTree at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!

You can see from the logged messages that the FormThanksPage is in a IOBucket which again is in a IOBTree which is in a object of the class five.intid.intid.IntIds which is part if the component-registry in the Plone site.

This means there is a reference to a broken object in the IntId tool. How to solve all these is covered below in the chapter "Frequent Culprits".

Decide how and if to fix it

In this case the solution is clear (remove refs to broken objects from the intid tool). But that is only one approach.

Often the solution is not presented like this (the solution to intid was not obvious to me until I spent considerable time to investigate).

The following six options to deal with these problems exists. Spoiler: Option 6 is the best approach in most cases but the other also have valid use-cases.

Option 1: Ignoring the errors

I do that a lot. Especially old databases that were migrated all the may from Plone 2 or 3 up to the current version have issues. If these issues never appear during operation and if clients have no budget or interest in fixing them you can leave them be. If they do not hurt you (e.g. you cannot pack your database or features actually fail) you can choose to ignore them.

At some point later they might appear and it may be a better time to fix them. I spent many hours fixing issues that will never show during operation.

Option 2: Migrating/Fixing a DB with zodbupdate

Use that when a module or class has moved or was renamed.


You can change objects in DB according to rules:

  • When a import has moved use a rename mapping
  • To specify if a obj needs to be decoded decode mapping

Examples from Zope/src/OFS/

zodbupdate_decode_dict = {
    'OFS.Image File data': 'binary',
    'OFS.Image Image data': 'binary',

    'OFS.Application Application title': 'utf-8',
    'OFS.DTMLDocument DTMLDocument title': 'utf-8',
    'OFS.DTMLMethod DTMLMethod title': 'utf-8',
    'OFS.DTMLMethod DTMLMethod raw': 'utf-8',
    'OFS.Folder Folder title': 'utf-8',
    'OFS.Image File title': 'utf-8',
    'OFS.Image Image title': 'utf-8',
    'OFS.Image Pdata title': 'utf-8',
    'OFS.Image Pdata data': 'binary',
    'OFS.OrderedFolder OrderedFolder title': 'utf-8',
    'OFS.userfolder UserFolder title': 'utf-8',

zodbupdate_rename_dict = {
    'webdav.LockItem LockItem': 'OFS.LockItem LockItem',

You can specify your own mappings in your own packages. These mappings need to be registered in so zodbupdate will pick them up.

Rename mapping example:

Decode mapping example:

Option 3: Work around with a patch

You can inject a module to work around missing or moved classes or modules.

The reason to want do this is usually because then you can safely delete items after that. They don't hurt your performance.

Examples in

# -*- coding: utf-8 -*-
from OFS.SimpleItem import SimpleItem
from import alias_module
from import bbb
from zope.interface import Interface

class IBBB(Interface):

class BBB(object):

SlideshowDescriptor = SimpleItem

# Interfaces
    from collective.z3cform.widgets.interfaces import ILayer
except ImportError:
    alias_module('collective.z3cform.widgets.interfaces.ILayer', IDummy)

    from App.interfaces import IPersistentExtra
except ImportError:
    alias_module('App.interfaces.IPersistentExtra', IDummy)

    from webdav.interfaces import IDAVResource
except ImportError:
    alias_module('webdav.interfaces.IDAVResource', IDummy)

# SimpleItem
    from collective.easyslideshow.descriptors import SlideshowDescriptor
except ImportError:
    alias_module('collective.easyslideshow.descriptors.SlideshowDescriptor', SlideshowDescriptor)

# object
    from collective.solr import interfaces
except ImportError:
    alias_module('collective.solr.indexer.SolrIndexProcessor', BBB)

    from Products.CMFPlone import UndoTool
except ImportError:
    sys.modules['Products.CMFPlone.UndoTool'] = bbb


Plone has plenty of these (see

Option 4: Replace broken objects with a dummy

If a objects is missing (i.e. you get a POSKeyError) or broken beyond repair you can choose to replace it with a dummy.

from persistent import Persistent
from ZODB.utils import p64
import transaction

app = self.context.__parent__
broken_oids = [0x2c0ab6, 0x2c0ab8]

for oid in broken_oids:
    dummy = Persistent()
    dummy._p_oid = p64(oid)
    dummy._p_jar = app._p_jar
    app._p_jar._added[dummy._p_oid] = dummy

You shoud be aware that the missing or broken object will be gone forever after you didi this. So before you choose to go down this path you should try to find out what the object in question actually was.

Option 5: Remove broken objects from db

from persistent import Persistent
from ZODB.utils import p64
import transaction

app = self.context.__parent__
broken_oids = [0x2c0ab6, 0x2c0ab8]

for oid in broken_oids:
    root = connection.root()
    del app._p_jar[p64(oid)]

I'm not sure if that is a acceptable approach under any circumstance since this will remove the pickle but not all references to the object. It will probably lead to PosKeyErrors.

Option 6: Manual fixing

This is how you should deal with most problems.

The way to go

  1. Use zodbverify to get all broken objects
  2. Pick one error-type at a time
  3. Use zodbverify with -o <OID> -D to inspect one object and find out where that object is referenced
  4. If you use follow referenced by until you find where in the tree the object lives. zodbverify will try to do it for you.
  5. Remove or fix the object (using a upgrade-step, pdb or a rename mapping)

Find out which items are broken

The newest version of zodbverify has a feature to that does the same task we discussed in Example 1 for you. Until it is merged and released you need to use the branch show_references from the pull-request

When inspecting a individual oid zodbverify builds a dict of all references for reverse-lookup. Then it recursively follow the trail of references to referencing items up to the root. To prevent irrelevant and recursive entries it aborts after level 600 and at some root-objects because these usually references a lot and would clutter the result with irrelevant information.

The output should give you a pretty good idea where in the object-tree a item is actually located, how to access and fix it.

If 0x3b1d06 is the broken oid inspect it with zodbverify:

$ ./bin/instance zodbverify -o 0x3b1d06 -D

2020-08-24 12:19:32,441 INFO    [Zope:45][MainThread] Ready to handle requests
2020-08-24 12:19:32,442 INFO    [zodbverify:222][MainThread]
The object is 'obj'
The Zope instance is 'app'
[4] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/
-> pickle, state = storage.load(oid)

(Pdb++) obj
<BTrees.OIBTree.OITreeSet object at 0x110b97ac0 oid 0x3b1d06 in <Connection at 10c524040>>

(Pdb++) pp [i for i in obj]
[<InterfaceClass OFS.EtagSupport.EtagBaseInterface>,
 <class 'webdav.interfaces.IDAVResource'>,
 <InterfaceClass plone.dexterity.interfaces.IDexterityContent>,
 <SchemaClass plone.supermodel.model.Schema>]

The problem now is that obj has no __parent__ so you have no way of knowing what you're actually dealing with.

When you press c for continue zodbverify will proceed and load the pickle:

(Pdb++) c
2020-08-24 12:20:50,784 INFO    [zodbverify:68][MainThread]
Could not process <class 'BTrees.OIBTree.OITreeSet'> record 0x3b1d06 (b'\x00\x00\x00\x00\x00;\x1d\x06'):
2020-08-24 12:20:50,784 INFO    [zodbverify:69][MainThread] b'\x80\x03cBTrees.OIBTree\nOITreeSet\nq\x00.\x80\x03(cOFS.EtagSupport\nEtagBaseInterface\nq\x01cAcquisition.interfaces\nIAcquirer\nq\\nIAllowDiscussion\nq\x03czope.annotation.interfaces\nIAnnotatable\nq\x04czope.annotation.interfaces\nIAttributeAnnotatable\nq\x05cplone.uuid.interfaces\nIAttributeUUID\nq\x06cProducts.CMFDynamicViewFTI.interfaces\nIBrowserDefault\nq\x07cProducts.CMFCore.interfaces\nICatalogAware\nq\x08cProducts.CMFCore.interfaces\nICatalogableDublinCore\nq\tczope.location.interfaces\nIContained\nq\ncProducts.CMFCore.interfaces\nIContentish\nq\x0bcOFS.interfaces\nICopySource\nq\x0ccwebdav.interfaces\nIDAVResource\nq\rcplone.dexterity.interfaces\nIDexterityContent\nq\\nIDexterityHasRelations\nq\x0fcplone.dexterity.interfaces\nIDexterityItem\nq\\nIDexterityIterateAware\nq\x11cplone.dexterity.interfaces\nIDexteritySchema\nq\\nIDocument\nq\x13cProducts.CMFCore.interfaces\nIDublinCore\nq\x14cProducts.CMFCore.interfaces\nIDynamicType\nq\\nIExcludeFromNavigation\nq\x16cz3c.relationfield.interfaces\nIHasIncomingRelations\nq\x17cz3c.relationfield.interfaces\nIHasOutgoingRelations\nq\x18cz3c.relationfield.interfaces\nIHasRelations\nq\x19cplone.namedfile.interfaces\nIImageScaleTraversable\nq\x1acOFS.interfaces\nIItem\nq\\nIIterateAware\nq\x1ccplone.portlets.interfaces\nILocalPortletAssignable\nq\x1dczope.location.interfaces\nILocation\nq\x1ecOFS.interfaces\nIManageable\nq\x1fcProducts.CMFCore.interfaces\nIMinimalDublinCore\nq cProducts.CMFCore.interfaces\nIMutableDublinCore\nq!cProducts.CMFCore.interfaces\nIMutableMinimalDublinCore\nq"\nINameFromTitle\nq#cApp.interfaces\nINavigation\nq$cProducts.CMFCore.interfaces\nIOpaqueItemManager\nq%cAccessControl.interfaces\nIOwned\nq&cAccessControl.interfaces\nIPermissionMappingSupport\nq\'cpersistent.interfaces\nIPersistent\nq(cOFS.interfaces\nIPropertyManager\nq)\nIRelatedItems\nq*cAccessControl.interfaces\nIRoleManager\nq+cplone.contentrules.engine.interfaces\nIRuleAssignable\nq,cProducts.CMFDynamicViewFTI.interfaces\nISelectableBrowserDefault\nq-cOFS.interfaces\nISimpleItem\\nITableOfContents\nq/cOFS.interfaces\nITraversable\nq0cplone.uuid.interfaces\nIUUIDAware\nq1cProducts.CMFEditions.interfaces\nIVersioned\\nIVersioningSupport\nq3cProducts.CMFCore.interfaces\nIWorkflowAware\nq4cOFS.interfaces\nIWriteLock\nq5cOFS.interfaces\nIZopeObject\nq6czope.interface\nInterface\nq7cplone.dexterity.schema.generated\nPlone_0_Document\nq8cplone.supermodel.model\nSchema\nq9tq:\x85q;\x85q<\x85q=.'
2020-08-24 12:20:50,786 INFO    [zodbverify:70][MainThread] Traceback (most recent call last):
  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/", line 64, in verify_record
  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/", line 62, in find_class
    return super(Unpickler, self).find_class(modulename, name)
ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package

    0: \x80 PROTO      3
    2: (    MARK
    3: c        GLOBAL     'OFS.EtagSupport EtagBaseInterface'
   38: q        BINPUT     1
   40: c        GLOBAL     'Acquisition.interfaces IAcquirer'
   74: q        BINPUT     2
   76: c        GLOBAL     ' IAllowDiscussion'
  135: q        BINPUT     3
  137: c        GLOBAL     'zope.annotation.interfaces IAnnotatable'
  178: q        BINPUT     4
  180: c        GLOBAL     'zope.annotation.interfaces IAttributeAnnotatable'
  230: q        BINPUT     5
  232: c        GLOBAL     'plone.uuid.interfaces IAttributeUUID'
  270: q        BINPUT     6
  272: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces IBrowserDefault'
  327: q        BINPUT     7
  329: c        GLOBAL     'Products.CMFCore.interfaces ICatalogAware'
  372: q        BINPUT     8
  374: c        GLOBAL     'Products.CMFCore.interfaces ICatalogableDublinCore'
  426: q        BINPUT     9
  428: c        GLOBAL     'zope.location.interfaces IContained'
  465: q        BINPUT     10
  467: c        GLOBAL     'Products.CMFCore.interfaces IContentish'
  508: q        BINPUT     11
  510: c        GLOBAL     'OFS.interfaces ICopySource'
  538: q        BINPUT     12
  540: c        GLOBAL     'webdav.interfaces IDAVResource'
  572: q        BINPUT     13
  574: c        GLOBAL     'plone.dexterity.interfaces IDexterityContent'
  620: q        BINPUT     14
  622: c        GLOBAL     ' IDexterityHasRelations'
  681: q        BINPUT     15
  683: c        GLOBAL     'plone.dexterity.interfaces IDexterityItem'
  726: q        BINPUT     16
  728: c        GLOBAL     ' IDexterityIterateAware'
  791: q        BINPUT     17
  793: c        GLOBAL     'plone.dexterity.interfaces IDexteritySchema'
  838: q        BINPUT     18
  840: c        GLOBAL     ' IDocument'
  885: q        BINPUT     19
  887: c        GLOBAL     'Products.CMFCore.interfaces IDublinCore'
  928: q        BINPUT     20
  930: c        GLOBAL     'Products.CMFCore.interfaces IDynamicType'
  972: q        BINPUT     21
  974: c        GLOBAL     ' IExcludeFromNavigation'
 1040: q        BINPUT     22
 1042: c        GLOBAL     'z3c.relationfield.interfaces IHasIncomingRelations'
 1094: q        BINPUT     23
 1096: c        GLOBAL     'z3c.relationfield.interfaces IHasOutgoingRelations'
 1148: q        BINPUT     24
 1150: c        GLOBAL     'z3c.relationfield.interfaces IHasRelations'
 1194: q        BINPUT     25
 1196: c        GLOBAL     'plone.namedfile.interfaces IImageScaleTraversable'
 1247: q        BINPUT     26
 1249: c        GLOBAL     'OFS.interfaces IItem'
 1271: q        BINPUT     27
 1273: c        GLOBAL     ' IIterateAware'
 1317: q        BINPUT     28
 1319: c        GLOBAL     'plone.portlets.interfaces ILocalPortletAssignable'
 1370: q        BINPUT     29
 1372: c        GLOBAL     'zope.location.interfaces ILocation'
 1408: q        BINPUT     30
 1410: c        GLOBAL     'OFS.interfaces IManageable'
 1438: q        BINPUT     31
 1440: c        GLOBAL     'Products.CMFCore.interfaces IMinimalDublinCore'
 1488: q        BINPUT     32
 1490: c        GLOBAL     'Products.CMFCore.interfaces IMutableDublinCore'
 1538: q        BINPUT     33
 1540: c        GLOBAL     'Products.CMFCore.interfaces IMutableMinimalDublinCore'
 1595: q        BINPUT     34
 1597: c        GLOBAL     ' INameFromTitle'
 1642: q        BINPUT     35
 1644: c        GLOBAL     'App.interfaces INavigation'
 1672: q        BINPUT     36
 1674: c        GLOBAL     'Products.CMFCore.interfaces IOpaqueItemManager'
 1722: q        BINPUT     37
 1724: c        GLOBAL     'AccessControl.interfaces IOwned'
 1757: q        BINPUT     38
 1759: c        GLOBAL     'AccessControl.interfaces IPermissionMappingSupport'
 1811: q        BINPUT     39
 1813: c        GLOBAL     'persistent.interfaces IPersistent'
 1848: q        BINPUT     40
 1850: c        GLOBAL     'OFS.interfaces IPropertyManager'
 1883: q        BINPUT     41
 1885: c        GLOBAL     ' IRelatedItems'
 1933: q        BINPUT     42
 1935: c        GLOBAL     'AccessControl.interfaces IRoleManager'
 1974: q        BINPUT     43
 1976: c        GLOBAL     'plone.contentrules.engine.interfaces IRuleAssignable'
 2030: q        BINPUT     44
 2032: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces ISelectableBrowserDefault'
 2097: q        BINPUT     45
 2099: c        GLOBAL     'OFS.interfaces ISimpleItem'
 2127: q        BINPUT     46
 2129: c        GLOBAL     ' ITableOfContents'
 2196: q        BINPUT     47
 2198: c        GLOBAL     'OFS.interfaces ITraversable'
 2227: q        BINPUT     48
 2229: c        GLOBAL     'plone.uuid.interfaces IUUIDAware'
 2263: q        BINPUT     49
 2265: c        GLOBAL     'Products.CMFEditions.interfaces IVersioned'
 2309: q        BINPUT     50
 2311: c        GLOBAL     ' IVersioningSupport'
 2370: q        BINPUT     51
 2372: c        GLOBAL     'Products.CMFCore.interfaces IWorkflowAware'
 2416: q        BINPUT     52
 2418: c        GLOBAL     'OFS.interfaces IWriteLock'
 2445: q        BINPUT     53
 2447: c        GLOBAL     'OFS.interfaces IZopeObject'
 2475: q        BINPUT     54
 2477: c        GLOBAL     'zope.interface Interface'
 2503: q        BINPUT     55
 2505: c        GLOBAL     'plone.dexterity.schema.generated Plone_0_Document'
 2556: q        BINPUT     56
 2558: c        GLOBAL     'plone.supermodel.model Schema'
 2589: q        BINPUT     57
 2591: t        TUPLE      (MARK at 2)
 2592: q    BINPUT     58
 2594: \x85 TUPLE1
 2595: q    BINPUT     59
 2597: \x85 TUPLE1
 2598: q    BINPUT     60
 2600: \x85 TUPLE1
 2601: q    BINPUT     61
 2603: .    STOP
highest protocol among opcodes = 2

If you are into this you can read the pickle now :)

If you press c again zodbverify will build the refernce-tree for this object and ispect if for you:

(Pdb++) c
2020-08-24 12:22:42,596 INFO    [zodbverify:234][MainThread] ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 0x3b1d06
2020-08-24 12:22:42,597 INFO    [zodbverify:43][MainThread] Building a reference-tree of ZODB...
2020-08-24 12:22:42,964 INFO    [zodbverify:60][MainThread] Objects: 10000
2020-08-24 12:22:44,167 INFO    [zodbverify:60][MainThread] Objects: 20000
2020-08-24 12:22:44,521 INFO    [zodbverify:60][MainThread] Objects: 30000
2020-08-24 12:22:44,891 INFO    [zodbverify:60][MainThread] Objects: 40000
2020-08-24 12:22:45,184 INFO    [zodbverify:60][MainThread] Objects: 50000
2020-08-24 12:22:45,507 INFO    [zodbverify:60][MainThread] Objects: 60000
2020-08-24 12:22:45,876 INFO    [zodbverify:60][MainThread] Objects: 70000
2020-08-24 12:22:46,403 INFO    [zodbverify:60][MainThread] Objects: 80000
2020-08-24 12:22:46,800 INFO    [zodbverify:60][MainThread] Objects: 90000
2020-08-24 12:22:47,107 INFO    [zodbverify:60][MainThread] Objects: 100000
2020-08-24 12:22:47,440 INFO    [zodbverify:60][MainThread] Objects: 110000
2020-08-24 12:22:47,747 INFO    [zodbverify:60][MainThread] Objects: 120000
2020-08-24 12:22:48,052 INFO    [zodbverify:60][MainThread] Objects: 130000
2020-08-24 12:22:48,375 INFO    [zodbverify:60][MainThread] Objects: 140000
2020-08-24 12:22:48,665 INFO    [zodbverify:60][MainThread] Objects: 150000
2020-08-24 12:22:48,923 INFO    [zodbverify:60][MainThread] Objects: 160000
2020-08-24 12:22:49,037 INFO    [zodbverify:61][MainThread] Created a reference-dict for 163955 objects.

2020-08-24 12:22:49,386 INFO    [zodbverify:182][MainThread] Save reference-cache as /Users/pbauer/.cache/zodbverify/zodb_references_0x03d7f331f3692266.json
2020-08-24 12:22:49,424 INFO    [zodbverify:40][MainThread] The oid 0x3b1d06 is referenced by:

0x3b1d06 (BTrees.OIBTree.OITreeSet) is referenced by 0x3b1d01 (BTrees.OOBTree.OOBucket) at level 1
0x3b1d01 (BTrees.OOBTree.OOBucket) is referenced by 0x11c284 (BTrees.OOBTree.OOBTree) at level 2
0x11c284 (BTrees.OOBTree.OOBTree) is _reltoken_name_TO_objtokenset for 0x11c278 (z3c.relationfield.index.RelationCatalog) at level 3
0x11c278 (z3c.relationfield.index.RelationCatalog) is relations for 0x1e (five.localsitemanager.registry.PersistentComponents) at level 4
0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x11 (Products.CMFPlone.Portal.PloneSite) at level 5
0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x01 (OFS.Application.Application) at level 6
8< --------------- >8 Stop at root objects
0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f6 (persistent.mapping.PersistentMapping) at level 7
8< --------------- >8 Stop at root objects
0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f7 (zope.component.persistentregistry.PersistentAdapterRegistry) at level 8
8< --------------- >8 Stop at root objects
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02f5 ( at level 6
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fa (zope.ramcache.ram.RAMCache) at level 7
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fd ( at level 8
0x02fd ( is __parent__ for 0x338f13 ( at level 9
0x338f13 ( is ++rule++rule-2 for 0x0303 (BTrees.OOBTree.OOBTree) at level 10
0x02fd ( is __parent__ for 0x346961 ( at level 10
0x02fd ( is __parent__ for 0x346b59 ( at level 11
0x02fd ( is __parent__ for 0x346b61 ( at level 12
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fe ( at level 9
0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x034d (plone.keyring.keyring.Keyring) at level 10
0x034d (plone.keyring.keyring.Keyring) is referenced by 0x02fb (persistent.mapping.PersistentMapping) at level 11
0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a32 (plone.keyring.keyring.Keyring) at level 12
0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a33 (plone.keyring.keyring.Keyring) at level 13
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x3b3dc4 (pas.plugins.ldap.plonecontrolpanel.cache.CacheSettingsRecordProvider) at level 11
0x3b1d01 (BTrees.OOBTree.OOBucket) is _next for 0x3b1cf1 (BTrees.OOBTree.OOBucket) at level 3

From this output you can find out that the broken object is (surprise) a item in the RelationCatalog of zc.relation. See the chapter "Frequent Culprits" for information how to deal with these.

Example 1 of using

In this and the next example I will use the script to find out where a broken objects actually sits so I can remove or fix it. The easier approach is to use zodbverify but I discuss this approach here since it was your best option until I extended zodbverify and since it might help you to understand the way references work in the ZODB.

$ ./bin/zodbverify -f var/filestorage/Data.fs

INFO:zodbverify:Done! Scanned 120797 records.
Found 116 records that could not be loaded.
Exceptions and how often they happened:
AttributeError: Cannot find dynamic object factory for module plone.dexterity.schema.generated: 20
AttributeError: module '' has no attribute 'IEventSettings': 3
ModuleNotFoundError: No module named 'Products.ATContentTypes': 4
ModuleNotFoundError: No module named 'Products.Archetypes': 5
ModuleNotFoundError: No module named 'Products.CMFDefault': 20
ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1
ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1
ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1
ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1
ModuleNotFoundError: No module named 'fourdigits': 8
ModuleNotFoundError: No module named '': 2
ModuleNotFoundError: No module named ''; '' is not a package: 34
ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

Follow the white rabbit...

./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x35907d

oid 0x35907d BTrees.OIBTree.OISet 1 revision
    tid 0x03c425bfb4d8dcaa offset=282340 2017-12-15 10:07:42.386043
        tid user=b'Plone'
        tid description=b'/Plone/it-service/hilfestellungen-anleitungen-faq/outlook/content-checkout'
        new revision BTrees.OIBTree.OISet at 282469
    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413
        tid user=b''
        tid description=b''
        referenced by 0x35907b BTrees.OIBTree.OITreeSet at 911946038


Follow referenced by ...

./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x35907b

referenced by 0x3c5790 BTrees.OOBTree.OOBucket
./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x3c5790

referenced by 0x11c284 BTrees.OOBTree.OOBTree
./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x11c284

referenced by 0x3d0bd6 BTrees.OOBTree.OOBucket
./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x3d0bd6

referenced by 0x11c278 z3c.relationfield.index.RelationCatalog

Found it!!!!!

Example 2 of using

In this example zodbverify found a trace of Products.PloneFormGen even though you think you safely uninstalled the addon (e.g. using

Then find out where exists in the tree by following the trail of items that reference it:

./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x372d00
oid 0x372d00 Products.PloneFormGen.content.thanksPage.FormThanksPage 1 revision
    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413
        tid user=b''
        tid description=b''
        new revision Products.PloneFormGen.content.thanksPage.FormThanksPage at 912841984
        referenced by 0x372f26 BTrees.OIBTree.OIBucket at 912930339
        references 0x372e59 Products.Archetypes.BaseUnit.BaseUnit at 912841984
        references 0x372e5a Products.Archetypes.BaseUnit.BaseUnit at 912841984
        references 0x372e5b OFS.Folder.Folder at 912841984
        references 0x372e5c Products.Archetypes.BaseUnit.BaseUnit at 912841984
        references 0x372e5d Products.Archetypes.BaseUnit.BaseUnit at 912841984
        references 0x372e5e Persistence.mapping.PersistentMapping at 912841984
    tid 0x03d40a3e52a41633 offset=921078960 2019-11-25 17:02:19.368976
        tid user=b'Plone pbauer'
        tid description=b'/Plone/rename_file_ids'
        referenced by 0x2c1b51 BTrees.IOBTree.IOBucket at 921653012

Follow referenced by until you find something...

./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x2c1b51
oid 0x2c1b51 BTrees.IOBTree.IOBucket 1 revision

Here I skip the trail of referenced by until I find 0x280184 five.intid.intid.IntIds:

./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x280184
oid 0x280184 five.intid.intid.IntIds 1 revision
    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413
        tid user=b''
        tid description=b''
        new revision five.intid.intid.IntIds at 8579054
        references 0x28018c <unknown> at 8579054
        references 0x28018d <unknown> at 8579054
    tid 0x03d3e90c4d3aed55 offset=915868610 2019-11-19 19:24:18.100824
        tid user=b' adminstarzel'
        tid description=b'/Plone/portal_quickinstaller/installProducts'
        referenced by 0x02f6 persistent.mapping.PersistentMapping at 915868690
        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394
        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394
        referenced by 0x1e five.localsitemanager.registry.PersistentComponents at 915898834

That is the IntId-Catalog from zope.intid. The problem seems to be that similar to the zc.relation catalog rerefences to broken objects stay in the catalog and need to be removed manually.

Here is a example of how to remove all broken objects from the catalog in a pdb-session:

(Pdb++) from zope.intid.interfaces import IIntIds
(Pdb++) from zope.component import getUtility
(Pdb++) intid = getUtility(IIntIds)
(Pdb++) broken_keys = [i for i in intid.ids if 'broken' in repr(i.object)]
(Pdb++) for broken_key in broken_keys: intid.unregister(broken_key)
(Pdb++) import transaction
(Pdb++) transaction.commit()

After packing the DB the problem is gone. o/

Other Options

Use zodbbrowser to inspect the ZODB.
It is Zope3 app to navigate a ZODB in a browser. At least I had problems getting it to run with a Plone-ZODB.
Use zc.zodbdgc
This tool can validate distributed databases by starting at their root and traversing to make sure all referenced objects are reachable. Optionally, a database of reference information can be generated.
Use collective.zodbdebug
A great tool to build and inspect reference-maps and backreference-maps of a ZODB. So for it does not work with Python 3 yet. Some if its features are also part of zodbverify.

Frequent Culprits

IntIds and Relations

The IntId-Tool and the relation-catalog are by far the most requent issues, especially if you migrated from Archetypes to Dexterity.

There may be a lot of RelationValues in these Tools that still reference objects that cannot be loadedif these removed objects were not properly removed.

The following code from collective.relationhelpers cleans up the IntId- and Relation-catalog but keeps relations intact. For large sites it may take a while to run because it also needs to recreate linkintegrity-relations.

from collective.relationhelpers.api import cleanup_intids
from collective.relationhelpers.api import purge_relations
from collective.relationhelpers.api import restore_relations
from collective.relationhelpers.api import store_relations

def remove_relations(context=None):
    # store all relations in a annotation on the portal
    # empty the relation-catalog
    # remove all relationvalues and refs to broken objects from intid
    # recreate all relations from a annotation on the portal

For details see


Many addons and features in Plone store data in Annotations on the portal or on content.

It's a good idea to check IAnnotations(portal).keys() after a migration for Annotation that you can safely remove.

Here is a example where wicked (the now-removed wiki-style-editing feature of Plone) stored it's settings in a Annotation:

def cleanup_wicked_annotation(context=None):
    ann = IAnnotations(portal)
    if '' in ann:
        del ann['']

Another example is files from failed uploads stored by plone.formwidget.namedfile in a annotation:

def cleanup_upload_annotation(context=None):
    # remove traces of aborted uploads
    ann = IAnnotations(portal)
    if ann.get('file_upload_map', None) is not None:
        for uuid in ann['file_upload_map']:
            del ann['file_upload_map'][uuid]


Migrating a ZODB from py2 to py3

Since people often encounter issues with their ZODB after migrating here is a quick dive into migrating a ZODB from Python 2 to Python 3.

The migration is basically calling the script zodbupdate in py3 with the parameter --convert-py3.

$ ./bin/zodbupdate --convert-py3

You need to pass it the location of the database, the defaul-encoding (utf8) and a fallback-encoding (latin1) for items where decoding to utf8 fails.


$ ./bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding=utf8 --encoding-fallback latin1

Updating magic marker for var/filestorage/Data.fs
Ignoring index for /Users/pbauer/workspace/projectx/var/filestorage/Data.fs
Loaded 2 decode rules from AccessControl:decodes
Loaded 12 decode rules from OFS:decodes
Loaded 2 decode rules from Products.PythonScripts:decodes
Loaded 1 decode rules from Products.ZopeVersionControl:decodes
Committing changes (#1).

After that you should be able to use your ZODB in Python 3.

The process in a nutshell:

  1. First, run bin/zodbupdate -f var/filestorage/Data.fs So no python3 convert stuff yet! This will detect and apply several explicit and implicit rename rules.
  2. Then run bin/instance zodbverify. If this still gives warnings or exceptions, you may need to define more rules and apply them with zodbupdate. But you can still choose to migrate to py3 if this shows errors.
  3. Using Python 3 run bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding utf8
  4. For good measure, on Python 3 run bin/instance zodbverify.

Read the docs:

See also:

Dealing with oids

Transforming oids from int to hex and text and vice versa:

>>> from ZODB.utils import p64
>>> oid = 0x2c0ab6
>>> p64(oid)

>>> from ZODB.utils import oid_repr
>>> oid = b'\x00\x00\x00\x00\x00,\n\xb6'
>>> oid_repr(oid)

>>> from ZODB.utils import repr_to_oid
>>> oid = '0x2c0ab6'
>>> repr_to_oid(oid)

Get a path for blobs:

from ZODB.blob import BushyLayout
if isinstance(oid, int):
    # e.g. oid = 0x2c0ab6
    from ZODB.utils import p64
    oid = p64(oid)
return BushyLayout.oid_to_path(None, oid)

Load a obj by oid from ZODB in a pdb-prompt:

oid = 0x2c0ab6
from ZODB.utils import p64
app = self.context.__parent__
obj = app._p_jar.get(p64(oid))


Finish, merge and document


Quite a lot of people from the Plone/Zope communities have wriotten about this issue. I learned a lot from these posts:

I got interviewed about Plone 5.2

erstellt von Philip Bauer zuletzt geändert: 2019-07-19T15:15:53+01:00
I got asked a bunch of more or less relevant questions regarding Plone, Python 3 and the future.

Click here to read the interview on

Four options to try Plone 5.2 on Python 3

erstellt von Philip Bauer zuletzt geändert: 2019-03-08T08:40:10+01:00
Try these different options to get started with Plone on Python 3.

Demo pages

There are nightly build of the current Plone 5.2 coredev (the development version) with Python 2 and 3:

Minimal Buildout

Here is a minimal buildout to run Plone 5.2rc1 on Python 3:

parts = instance
extends =

recipe = plone.recipe.zope2instance
eggs =

You set it up like this:

$ python3.7 -m venv .
$ ./bin/pip install -r
$ ./bin/buildout

And start it as usual with ./bin/instance fg

Standalone Development buildout

extends =

parts =

eggs =

test-eggs =
    collective.easyform [test]

auto-checkout =

extensions =

show-picked-versions = true

recipe = plone.recipe.zope2instance
user = admin:admin
eggs = ${buildout:eggs}
debug-mode = on
verbose-security = on

recipe = zc.recipe.egg
eggs =
interpreter = zopepy
scripts =

recipe = collective.recipe.omelette
ignore-develop = False
eggs = ${buildout:eggs}
ignores = roman

recipe = collective.xmltestreport
eggs = ${buildout:test-eggs}
defaults = ['--auto-color', '--auto-progress']

recipe = zc.recipe.egg
eggs =

collective.easyform = git branch=python3


Starzel buildout

The buildout that we at use supports Plone 5.2rc1 with Python 2 and 3.

It has some nice features:

  • It extends to config- and version-files on github shared by all projects that use the same version of Plone.
  • It allows to update a project simply by changing the version it extends.
  • It allows to update all projects of one version by changing remote files (very useful for HotFixes).
  • It is minimal work to setup a new project.
  • It has presets for development, testing, staging and production.
  • It has all the nice development-helpers we use.


$ git clone -b 5.2rc1.x <SOME_PROJECT>

Remove all files that are not needed for a project but are only used for the buildout itself.

$ rm -rf linkto README.rst README.txt .travis.yml secret.cfg_tmpl VERSION.txt local_coredev.cfg CHANGES.rst

If you're not developing the buildout itself you want a create a new git repo.

$ rm -rf .git && git init

Add a file that contains a passwort. Do not use admin as a password in production!

$ echo -e "[buildout]\nlogin = admin\npassword = admin" > secret.cfg

Symlink to the file that best fits you local environment. At first that is usually development. Later you can use production or test. This buildout only uses local.cfg and ignores all local_*.cfg.

$ ln -s local_develop.cfg local.cfg

Create a virtualenv in Python 2.7 or Python 3.7 (Plone 5.2 only).

$ virtualenv .  # for Python 2.7
$ python3.7 -m venv .  # for Python 3 (Plone 5.2 only)

Install and configure Plone

$ ./bin/pip install -r requirements.txt
$ ./bin/buildout


I hope these options help you get started with Python 3. For serious projects you will likely create you own buildout.

Update (4.2.2019)

You should use the Plone coredev for developing Plone itself:

$ git clone coredev
$ cd coredev
$ git checkout 5.2
$ python3.7 -m venv .
$ ./bin/pip install -r requirements.txt
$ ./bin/buildout
$ ./bin/instance fg

Update 2 (24.2.2019)

Since Plone 5.2b2 wsgi is the default, even when running Python 2. See

This means you do not need to enable wsgi = on since it will be enabled automatically. You can still choose to switch wsgi = off in Python 2 if you have a good reason to do so (e.g. you need FTP or WebDAV). I updated the examples accordingly.

Update 3 (8.3.2019)

Release candidate 1 of Plone 5.2 is pending. See I updated the examples accordingly.

Plone finally supports Python 3!

erstellt von Philip Bauer zuletzt geändert: 2018-11-29T22:41:55+01:00
Our favourite Open Source CMS has landed on Python 3. Here is what you need to know about migrating projects.

At the Plone Conference 2018 in Tokyo I gave a plenary talk in which I told the story how Plone got to support Python 3. Ten minutes before my talk Plone 5.2a1 was released. That release was the culmination of a 3-year long journey (or even 5 years if you start in Brazil). If you want to learn how we got there you should watch the talk.

Where are we now?

Plone 5.2 supports Python 3.7, 3.6 and 2.7. It is currently in alpha but the plan is to release a final version in February 2019 right after the Alpine City Sprint. You should seriously consider joining that sprint if you plan to migrate your own projects. Working together with the people who ported Plone to Python 3 will give you the know-how to succeed in your own migrations!

But in fact you should not wait until February to start working with Plone on Python 3 but start right now so you'll have enough time to migrate to Python 3 before January 1st 2020 (End of Life of Python 2).

To make testing and working with Python 3 easier some tools and addons were already working on Python 3.

The following essential development-tools are already ported:

Also the three addons that achieved the highest ranking during the latest poll at Ploneconf now work on Python 3:

You can also run the new frontend Volto with Plone on Python 3.

Open tasks

Before the final release of Plone 5.2 scheduled for February 2019 there are still a lot of things to do.

Here are the six most pressing issues:

  1. The upgrade- and porting-guide need to be completed.
  2. ZODB migration needs to be finished and documented.
  3. Much more addons need to be ported and released.
  4. We require performance tests.
  5. People who really need a replacement for FTP and/or WebDAV need to figure out how to do that.

There is still a lot to do. You can help Plone by coming to a sprint, testing it and fixing or reporting a bug.

Migrating to Python 3

From 1.1.2020 on you'll have to inform your clients that you are running their super-secure system on an unsupported version of Python. That might be ok if you're running a small website for a friend but for serious projects this simply will not do. Instead you should start to adopt Python 3 in your Plone projects now. You need to start early and plan ahead!

Here are the six steps you have to follow to upgrade to Python 3. Do not attempt to do all steps at once, instead you should work iteratively and even deploy to production in between steps.

1. Upgrade to Plone 5.2

You should still use Python 2.7 for this step. The changes between Plone 5.1 and Plone 5.2 are not huge but you will gain some great features including a new and much faster dropdown-navigation and a bootstrap-based ZMI.

The Upgrade-Guide for 5.2 still needs much more love. So if you encounter any issues please update that document and/or create a ticket in github.

2. Drop Archetypes and only use Dexterity

Archetypes is still supported on Plone 5.2 but only when running with Python 2.7. It is now officially deprecated and is not ported to Python 3. There has been a built-in migration for default types from Archetypes to Dexterity since 2013, and since 2015 there have been helpers and even a form that allows you to migrate your custom content.

3. Migrate your code to Python 3

Follow the documentation in to migrate your custom code and any addons that you need in your packages.

  • Make sure your code works in Python 2 and Python 3. If you are certain that you will not publish that code as a addon or reuse it in another project you can also drop support for Python 2. That will make porting and testing easier.
  • For most addons: Use python-modernize as described in the documentation.
  • For small addons you could even simply try to startup on Python 3 and and fix whatever fails (e.g. relative imports or invalid syntax)
  • Manually fix whatever python-modernize misses (e.g. relative imports)
  • Do the same with tests. You'll need a test-setup with tox and a test-matrix in travis that tests different Python versions. The setup in collective.ifttt is almost done and might be a good example.
  • If you have complex doctests consider migrating them to python tests since these are easier to get to pass in both Python 2 and 3. If you do want to keep them as doctests, change them so that the output for Python 3 is the default and use a Py23DocChecker to make the tests pass in Python 2 as well. See the example in plone.indexer.
  • Once all seems to work you need to test every single feature manually because even a high test coverage is never a guarantee that your code actually does what it should.

4. Migrate Addons and Dependencies

Now migrate any addons you use to Python 3 as explained above. The main difference is that now everyone who uses this addon benefits. You shoud not drop support for Python 2 for those!

I listed the addons that already run on Python 3 above. I'm sure that list will be outdated very soon. Before you start migrating an addon you need to check if there already is a branch that supports Python 3 and check if there is a ticket like the one for collective.easyform. If not create such a ticket and start the process.

5. Migrate your Database

ZODB itself is compatible with Python 3 but a database that was created in Python 2.7 cannot be used in Python 3 without being modified first. I will not go into details about that. You should read the documentation and see David Glick's talk.

The migration will probably require some downtime and in rare cases you might have to write your own mappings to tell zodbupdate how to handle your data.

6. Deploy on Python 3

Plone 5.2 will use WSGI with waitress as the default WSGI server. Alternatively you can use uwsgi or gunicorn. ZServer can only be used with Python 2.7.

Some deployment tools may not ported yet, others like ZRS or RelStorage already support Python 3. You should test your production setup early. There may be changes that you don't want to struggle with a week before a scheduled launch.

Summing up

Start that whole process as early as possible because you may need to deal with unexpected problems and addons that you did not know still used Archetypes. 2019 might be much busier for you than you expect.

Do not wait for a final release of Plone 5.2! Start the process of planning and testing a migration. Consinder hiring experts to help you if necessary.

Again: You can and you should work iteratively. That means you do not have to do everything in one step and one night:

  • Upgrade to 5.2 with Dexterity on Python 2 and deploy that early in 2019.
  • Then take some time to migrate and test your code and addons.
  • You can then deploy the Python 3-compatible code while still running Python 2.7.
  • Then you can migrate your existing database and deploy on Python 3.


Here are some questions you might have:

What about Python 4?

That will not be a big issue since 4.0 will be a upgrade like from 3.7 to 3.8. The Python community simply don't want to release a Python 3.10.

When will Plone drop support for Python 2?

Not before we have a good reason. Maybe Plone 7? It would allow us to remove and clean up some code but we would not gain a lot.

What about Zope 5?

There are no real plans for that yet.

Will Archetypes be migrated to Python 3?

Please don't try to postpone the inevitable. We are too small a community to maintain two Python versions, two frontends and two content type frameworks.

How do we replace FTP/WebDAV?

I don't know, for WebDAV you might be able to use a middleware like WsgiDAV.

Why did you make all that effort and not 'simply' migrate to Guillotina?

Guillotina does not aim to be a replacement for Plone. Also it has no equivalent addon ecosystem, feature equality or migration path.

Can I now use async/await?

You can but it won't really help you since Plone and Zope are still fundamentally synchronous.

Why doesn't the Plone Foundation simply buy support for Python 2.7 from RedHat/IBM for all of us?

Are you kidding me?

Does Python 3 make Plone run faster?

Maybe. We still lack performance tests but my guess is that Plone 5.2 will run faster on Python 3 than on 2.

Can I run the same Database in Python 2 and 3?

No.You can neither use a DB create with py2 in py3 nor use one created with py3 in py2.

Is addon xyz already ported?

Most likely not.

Plone supports the latest two major version. Does that extend to Python?

Plone's security update policy supports the latest two major versions (currently, that means the 5.x series and the 4.3.x series) with security updates. That guarantee does not extend to the programming language, the operating systen or the hardware you are running Plone on.

Will this work on Windows?

Very likely. Someone will have to test it.

I need help with migrating our projects to Python 3. Who can help me?

Talk to us at or ask other professionals with experience.

Warm words

Migrating Plone to Python 3 was quite a ride and a great community effort. I had to cope with some brain-melting changes in Zope, fight heartbreaking test-isolation issues and toil through an endless list of packages and tests. But I had the immense privilege and pleasure to do all of that together with some of the smartest and nicest people I know. Thanks to all of you!

Watch the full talk if you want to learn more about how Plone finally landed on Python 3.

Here are all videos from Ploneconf 2018: Day 1, Day 2, Day 3.

Python 3, Plone 5.2, Zope 4 and more

erstellt von Philip Bauer zuletzt geändert: 2018-10-19T10:14:08+01:00
Report from the SaltLabs Sprint in Halle, Germany and the aftermath

With over 30 attendees the SaltLabs Sprint was among the bigger sprints in the history of Plone and Zope.

Fortunately, our host Gocept just moved into a new office and had plenty of space to spare. Stylish as we are, the Plone crowd quickly congregated in the palace-like salon with red walls and renaissance paintings. Even better: Throughout the week the cafe Kaffej downstairs was open for us to eat breakfast, lunch and for work. We tried to keep the barista busy producing excellent Cappuccino. Thanks!

Zope 4

The sprint split in two groups, one working on Zope and one working on Plone. A short report on about what happened at the Zope can be found at . Zope 4.0b6 was released just after the sprint. The new Beta 6 includes the new bootstrap-styled ZMI, support for Python 3.7 and a metric ton of bug fixes compared to 4.0b5 which was released in May this year. By the way: Zope can now be installed without buildout using pipenv

Python 3

The Plone crowd was huge and worked on several tasks. The biggest task - and the main reason to join the sprint - was continuing to port Plone to Python 3. Jens Klein, Peter Mathis, Alessandro Pisa, David Glick (remote) and Philip Bauer as lead worked mainly on fixing all remaining failing tests and getting the jenkins-builds to pass. We dove deep into crazy test-isolation-issues but also added a couple of missing features to Dexterity that were so far only tested with Archetypes.

During the sprint we got very close to a green build but a couple of really nasty test-isolation issues prevented a green build. A week after the sprint we finally got that done and all tests of Plone now pass in Python 2.7, Python 3.6 and Python 3.7 with the same code-base!

With the test passing we're now merging that work and preparing the release of Plone 5.2a1. The tasks for this are discussed in this ticket

Deprecate Archetypes

Archetypes will not be ported to Python 3 but still has to work when running Plone 5.2 in Python 2.7. Since all tests that used PloneTestCase were still using Archetypes Philip changed that to use Dexterity now and created a new testlayer ( that is now used by all the packages using the Archetypes-stack. See You could use this layer in your archetypes-based addon-packages that you want to port to Plone 5.2.


Yes, Plone is now able to run on Python 3. But without the huge ecosystem of addons that would be of limited real-world use. Jan Mevissen, Franco Pellegrini (remote) and Philip Bauer ported a couple of addons to Python 3 that Plone developers use all the time:

Being able to run tests against Plone in Python 3, and with these development-tools now available also, it is now viable to start porting other addons to Python 3. To find out how hard that is I started porting collective.easyform. The work is only 97% finished since some tests need more work but it is now useable in Python 3!

ZODB Migration

Harald Friesenegger and Robert Buchholz worked on defining a default migration-story for existing databases using zodbupdate. To discuss the details of this approach we had a hangout with Jim Fulton, David Glick and Sylvain Viollon. They solved a couple of tricky issues and wrote enough documentation. This approach seems to work well enough and the documentation points out some caveats but the ZODB migration will require some more work.

Relevant links:

ZODB Issues

Sune Broendum Woeller worked on a very nasty and complex issue with KeyErrors raising while releasing resources of a connection when closing the ZODB. That happened pretty frequently in our test-suites but was so far unexplained. He analyzed the code and finally was able to add a failing test to prove his theory. Then Jim Fulton realized the problem and wrote a fix for it. This will allow us to update to the newest ZODB-version once it is released. See for details.


Thomas Schorr added the zconsole module to Zope for running scripts and an interactive mode. Using WSGI works in Python 2 and Python 3 and will replace ZServer in Python 3.

Frontend and Theming

Thomas Massman, Fred van Dijk, Johannes Raggam and Maik Derstappen looked into various front-end issues, mainly with the Barceloneta theme. They closed some obsolete tickets and fixed a couple of bugs.

They also fixed some structural issues within the Barceloneta theme. The generated HTML markup now has the correct order of the content columns (main, then left portlets, then right portlets) which allows better styling for mobile devices. Also in the footer area we are now able to add more portlets to generate a nice looking doormat. See the ticket for screenshots and details.

They also discussed a possible enhancement of the Diazo based theming experience by including some functionality of spirit.plone.theming into and cleaning up the Theming Control Panel. A PLIP will follow for that.

New navigation with dropdown support

Peter Holzer (remote) continued to work on the new nativation with dropdown-support which was started on collective.navigation by Johannes Raggam. Due to the new exclude_from_nav index and optimized data structures the new navigation is also much faster than other tree based navigation solutions (10x-30x faster based on some quick tests).

Static Resource Refactoring

Johannes Raggam finished work on his PLIP to restructure static resources. With this we no longer carry 60MB of static resources in the CMFPlone repository, it allows to use different versions of mockup in Plone, enable us to release our libraries on npm and will make it easier to switch to a different framework. Skin scripts removal Katja Süss, Thomas Lotze, Maurits van Rees and Manuel Reinhardt worked hard at removing the remaining python_scripts. The work is still ongoing and it would be great to get rid of the last of these before a final Plone 5.2 release. See for details. Katja also worked on finally removing the old resource registry (for js and css).

Test parallelization

Joni Orponen made a lot of progress on speeding up our test-runs by running different test-layers in parallel. The plan is to get them from 30-60 minutes (depending on server and test-setup) to less than 10 minutes. For a regularly updated status of this work see

Documentation and User-Testing

Paul Roeland fought with robot-tests creating the screenshots for our documentation. And won. A upgrade guide to Plone 5.2 and Python 3 was started by several people. Jörg Zell did some user-testing of Plone on Python 3 and documented some errors that need to be triaged.


Jörg Zell worked on the german translations for Plone and nearly got to 100%. After the sprint Katja Süß did an overall review of the german translation and found some wording issues with need for a discussion

Plone React

Rob Gietema and Roel Bruggink mostly worked on their trainings for React and Plone-react, now renamed to “Volto”. Both will be giving these trainings at Plone Conference in Tokyo. On the second day of the sprint Rob demoed the current state of the new react-based frontend for Plone.

Assorted highlights

  • In a commit from 2016 an invisible whitespace was added to the doctests of plone.api. That now broke our test in very obscure ways. Alessandro used some dark magic to search and destroy.
  • The __repr__ for persistent objects changed breaking a lot of doctests. We still have to figure out how to deal with that. See for details.
  • There was an elaborate setup to control the port during robot-tests. By not setting a port at all the OS actually takes care of this to makes sure the ports do not conflict. See Less is sometimes more.
  • Better late than never: We now have a method safe_nativestring in Products.CMFPlone.utils besides our all-time favorites safe_unicode and safe_encode. It transforms to string (which is bytes in Python 2 and text in Python 3). By the way: There is also zope.schema.NativeString and zope.schema.NativeStringLine.
  • We celebrated the 17th birthday of Plone with a barbecue and a generous helping of drinks.

Porting Plone to Python 3

erstellt von Philip Bauer zuletzt geändert: 2018-05-24T09:40:11+01:00
&tldr; Plone is running nicely on Python 3 but now we have to fix over 500 failing tests.

Since I wrote the proposal to Port Plone to Python 3, so much has happened that a status update is needed.

Let's step back a little: The first steps towards Python 3 were taken during the sprint at the Plone conference in Barcelona. The epic PLIP to update to Zope 4 was merged and we started porting individual packages to Python 3 without being able to run any tests. With the help of sixer and python-modernize we tried to get the most obvious import and syntax issues out of the way. That work was continued at the Alpine City Sprint in Innsbruck and in the end, we treated more than 150 packages like this. Some of this work is documented in the Plone tracker.

Along the way I wrote a best-practice guide on how to port Plone packages this way.

As the PLIP states, there are several approaches to porting Plone to Python 3:

  1. Migrate and test packages that have no dependency to CMFPlone and test them in Python 2 and 3.
  2. Prepare packages for Python 3 without being able to test them in Python 3.
  3. Start up Plone on Python 3 and fix whatever breaks. When start-up works, create a Plone Site and again, fix whatever breaks.
  4. Port plone.testing and to Python 3 and start running tests. Fix what breaks during the setup of the layers.
  5. Run the tests with Python 3 and fix all broken tests.

At the sprint in Innbruck I started with #3 and kept going after the sprint until I was able to create an instance. At the Plone Tagung in Berlin I was able to demo the creation of a site but nothing was rendered yet.

After that, I kept going and I was finally able to create a site and manage content, which is what a CMS is about. It looked a bit raw but I was able to add and edit some content - yay!

early screenshot of plone with python 3

Work continued at an unsteady pace, and with important contributions from Michael Howitz, Alessandro Pisa and David Glick, things started to get better. Even the theme and js started working. Slowly broken features became the exception, not the rule.

Last week at the Zope 4 Welcome Sprint in Halle we removed all feature blockers that we had found so far. Plone with Python 3 looks and feels like it used to in Python 2. During the sprint there was also a lot of progress on:

  • the wsgi setup
  • logging and tracebacks when using wsgi
  • porting plone.testing
  • a new theme for the ZMI
  • beta releases for many packages.

There was also some progress on the difficult issue of database migrations. It seems like zodbupdate is the best tool to do that but there is probably a lot of work ahead.

Read more about the sprints outcome in the blogpost by Michael Howitz.

There is a Jenkins job and as of today, it is running (not passing) tests for all packages (except Archetypes) with Python 3.

At the moment we run 6594 tests from 115 packages with 257 failures and 315 errors - not bad when you keep in mind that we still focus on features, not on tests. Tests for a couple of packages are green, including plone.api which is great since plone.api uses most of our stack one way or another. Jenkins runs every three hours and here you can see the progress over time.

Here is a screenshot from today that shows Plone doing some pretty non-trivial things: Editing a recurring Event and also Richtext with Images and Links:

screenshot today

Next steps:

  • Create a demo site running on Python 3 that people can use to find broken features. This will happen at the Plonator Sprint in Munich.
  • Fix all tests that look like they fail because of a broken feature.
  • Fix all remaining tests to pass in Python 2 and Python 3. Since there are a gazillion doctests that will take some time.
  • Port to python 3 and fix robottests.
  • Experiment with porting a ZODB with Plone to Python 3.

If you want to participate you can simply look at the failing tests and start fixing them. You can also try to fix one of the open issues.
The setup of a coredev environment with Python 3 is really simple and documented.

Obstacles on the road towards Plone 2020

erstellt von Philip Bauer zuletzt geändert: 2017-11-10T10:54:41+01:00
A short story about debugging issues with test isolation

During the sprint at the Plone Conference 2017 in Barcelona, Plone achieved a major milestone towards what is often called "Plone 2020". This is basically the effort to modernize Plone's backend and achieve Python 3 compatibility. In 2020, support for Python 2.7 will officially end, hence Plone 2020.

A necessary part of that effort was to migrate Zope to Python 3, a daunting task that was only possible by a flurry of activity that combined the efforts of many stakeholders (not only the Plone Community). Learn more about that in Hanno Schlichting's talk once the video is on the website, and on many blog posts on the Gocept Blog.

Getting Plone to run on that newest version of Zope (currently Zope 4.0b2) was another story and took a lot of work (some details are in my post here. Finally in Barcelona, in a daring move we merged all the work that had been done for that PLIP and decided that the result will be called Plone 5.2. But by that time not all tests were green (that's why it was daring). We worked hard to get the tests to pass and to fix some issues we found when testing manually.

By the way: At the same sprint we started to prepare Plone itself for Python 3 by fixing all imports to work in both Python 2 and Python 3. But that is a tale for another blog post.

So, despite out best efforts, even one week after the conference I was not yet able to fix all the tests, and so I created at ticket to track the remaining issues.

Here this story about two erroring tests in Products.CMFFormController actually begins. Here is the spoiler: I did not really solve the issue but finally worked around it. But I still think the approach I took might be of interest to some.

The two breaking tests, test_attacker_redirect and test_regression, were passing when I ran them in isolation or when I ran all test of Products.CMFFormController with ./bin/test -s Products.CMFFormController. To add insult to injury, Products.CMFFormController is basically dead code but is still used by some of our legacy ControllerPageTemplates.

So how could I find the issue since the traceback was not really helpful?

Here is the relevant part of the log from jenkins:

#### Running tests for group Archetypes ####
Running Products.Archetypes.tests.attestcase.Archetypes:Functional tests:


Running tests:
  Tear down Testing.ZopeTestCase.layer.ZopeLite in 0.000 seconds.
  Set up plone.testing.zca.LayerCleanup in 0.000 seconds.
  Set up plone.testing.z2.Startup in 0.101 seconds.
  Set up in 9.722 seconds.
  Set up in 2.628 seconds.
  Set up in 0.000 seconds.

Error in test test_attacker_redirect (Products.CMFFormController.tests.testRedirectTo.TestRedirectToFunctional)
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/", line 329, in run
  File "/home/jenkins/workspace/plone-5.2-python-2.7-at/src/Products.CMFFormController/Products/CMFFormController/tests/", line 97, in test_attacker_redirect
  File "/home/jenkins/workspace/plone-5.2-python-2.7-at/src/Zope/src/Testing/ZopeTestCase/", line 43, in wrapped_func
    return func(*args, **kw)
  File "/home/jenkins/workspace/plone-5.2-python-2.7-at/src/Zope/src/Testing/ZopeTestCase/", line 127, in publish
    wsgi_result = publish(env, start_response)
  File "/home/jenkins/workspace/plone-5.2-python-2.7-at/src/Zope/src/ZPublisher/", line 254, in publish_module
    with load_app(module_info) as new_mod_info:
  File "/usr/lib/python2.7/", line 17, in __enter__
  File "/home/jenkins/workspace/plone-5.2-python-2.7-at/src/Zope/src/Testing/ZopeTestCase/", line 73, in load_app
    with ZPublisher.WSGIPublisher.__old_load_app__(module_info) as ret:
  File "/usr/lib/python2.7/", line 17, in __enter__
  File "/home/jenkins/workspace/plone-5.2-python-2.7-at/src/Zope/src/ZPublisher/", line 220, in load_app
    app = app_wrapper()
  File "/home/jenkins/workspace/plone-5.2-python-2.7-at/src/Zope/src/App/", line 78, in __call__
    return connection.root()[self._name]
  File "/home/jenkins/shiningpanda/jobs/2fa08faf/virtualenvs/d41d8cd9/lib/python2.7/", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'Application'

Error in test test_regression (Products.CMFFormController.tests.testRedirectTo.TestRedirectToFunctional)
Traceback (most recent call last):


    raise KeyError(key)
KeyError: 'Application'

  Ran 68 tests with 0 failures, 2 errors and 0 skipped in 1.626 seconds.
Running testing:Integration tests:
  Set up in 0.027 seconds.
  Set up testing:Integration in 0.000 seconds.
  Ran 27 tests with 0 failures, 0 errors and 0 skipped in 9.033 seconds.


Tearing down left over layers:
  Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.
Total: 733 tests, 0 failures, 2 errors and 0 skipped in 3 minutes 10.739 seconds.
#### Finished tests for group Archetypes ####

What? Why does connection.root() have no Application? This makes no sense to me, and a pdb there did not help to shed light on it at all.

First I reproduced the error by testing all packages in the test group Archetypes (where the error occurs):

./bin/test \
  -s Products.Archetypes \
  -s Products.CMFFormController \
  -s Products.MimetypesRegistry \
  -s Products.PortalTransforms \
  -s Products.statusmessages \
  -s Products.validation \

Then I only used the test layers that actually got set up according to the output:

./bin/test --layer Products.Archetypes.tests.attestcase.Archetypes \
           --layer Products.PortalTransforms.testing.PortalTransformsLayer \
           --layer Testing.ZopeTestCase.layer.ZopeLite \
           --layer \
           -s Products.Archetypes \
           -s Products.CMFFormController \
           -s Products.MimetypesRegistry \
           -s Products.PortalTransforms \
           -s Products.statusmessages \
           -s Products.validation \

That worked, I see the error. But I will not try to read 733 tests and wait for more than 3 minutes each time I think I may have fixed something!

Thus I used the divide-and-conquer strategy to figure out which combination produced the failing tests: remove half of the packages layers and see if it still fails. If they pass, try the other half. Do the same with the layers.

Remember to keep --layer and -s Products.CMFFormController in order not to skip the tests that expose the issue.

It turned out that the following combination reproduced the issue:

./bin/test \
    --layer Products.Archetypes.tests.attestcase.Archetypes \
    --layer Testing.ZopeTestCase.layer.ZopeLite \
    --layer \
    -s Products.Archetypes \
    -s Products.CMFFormController

Still way too many tests to have a look, most of them in Products.Archetypes. So I removed (actually, moved the .py files to some temp folder) all python tests and kept the doctests (and their setup). The only reason was that I hate doctests and consequently it must be a doctest that created trouble. I was right.

So I kept only one doctest that produced the issue by commenting out the others in of Products.Archetypes.

Now I needed to find a combination of three tests from these layers that still exposed the issue. To to that, I added the option -vv to the testrunner to see the names and python path of all tests that still ran.

./bin/test --layer Products.Archetypes.tests.attestcase.Archetypes --layer Testing.ZopeTestCase.layer.ZopeLite --layer -s Products.Archetypes -s Products.CMFFormController -vv
Running tests at level 1
Running Products.Archetypes.tests.attestcase.Archetypes:Functional tests:
  Set up plone.testing.zca.LayerCleanup in 0.000 seconds.
  Set up plone.testing.z2.Startup in 0.157 seconds.
  Set up in 10.252 seconds.
  Set up in 1.871 seconds.
  Set up Products.Archetypes.tests.attestcase.ATTestCaseFixture in 0.647 seconds.
  Set up Products.Archetypes.tests.attestcase.Archetypes:Functional in 0.000 seconds.
    1/1 (100.0%) /Users/pbauer/workspace/coredev/src/Products.Archetypes/Products/Archetypes/tests/traversal_4981.txt

  Ran 1 tests with 0 failures, 0 errors, 0 skipped in 0.269 seconds.
Running Testing.ZopeTestCase.layer.ZopeLite tests:
  Tear down Products.Archetypes.tests.attestcase.Archetypes:Functional in 0.000 seconds.
  Tear down Products.Archetypes.tests.attestcase.ATTestCaseFixture in 0.010 seconds.
  Tear down in 0.009 seconds.
  Tear down in 0.065 seconds.
  Tear down plone.testing.z2.Startup in 0.004 seconds.
  Tear down plone.testing.zca.LayerCleanup in 0.001 seconds.
  Set up Testing.ZopeTestCase.layer.ZopeLite in 0.009 seconds.
    1/5 (20.0%) test_parseXML_empty (Products.CMFFormController.tests.test_exportimport.CMFFormControllerImportConfiguratorTests)
    2/5 (40.0%) test_parseXML_with_info (Products.CMFFormController.tests.test_exportimport.CMFFormControllerImportConfiguratorTests)
    3/5 (60.0%) test_action_not_unicode (Products.CMFFormController.tests.test_exportimport.Test_importCMFFormController)
    4/5 (80.0%) test_normal (Products.CMFFormController.tests.test_exportimport.Test_importCMFFormController)
    5/5 (100.0%) test_partial (Products.CMFFormController.tests.test_exportimport.Test_importCMFFormController)

  Ran 5 tests with 0 failures, 0 errors, 0 skipped in 0.023 seconds.
Running tests:
  Tear down Testing.ZopeTestCase.layer.ZopeLite in 0.000 seconds.
  Set up plone.testing.zca.LayerCleanup in 0.000 seconds.
  Set up plone.testing.z2.Startup in 0.092 seconds.
  Set up in 7.227 seconds.
  Set up in 2.087 seconds.
  Set up in 0.000 seconds.
    1/4 (25.0%) testCopy (Products.CMFFormController.tests.testCopyRename.TestCopyRename)
    2/4 (50.0%) testRename (Products.CMFFormController.tests.testCopyRename.TestCopyRename)
    3/4 (75.0%) test_attacker_redirect (Products.CMFFormController.tests.testRedirectTo.TestRedirectToFunctional)

Error in test test_attacker_redirect (Products.CMFFormController.tests.testRedirectTo.TestRedirectToFunctional)
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/", line 329, in run
  File "/Users/pbauer/workspace/coredev/src/Products.CMFFormController/Products/CMFFormController/tests/", line 97, in test_attacker_redirect
  File "/Users/pbauer/workspace/coredev/src/Zope/src/Testing/ZopeTestCase/", line 43, in wrapped_func
    return func(*args, **kw)
  File "/Users/pbauer/workspace/coredev/src/Zope/src/Testing/ZopeTestCase/", line 127, in publish
    wsgi_result = publish(env, start_response)
  File "/Users/pbauer/workspace/coredev/src/Zope/src/ZPublisher/", line 254, in publish_module
    with load_app(module_info) as new_mod_info:
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/", line 17, in __enter__
  File "/Users/pbauer/workspace/coredev/src/Zope/src/Testing/ZopeTestCase/", line 73, in load_app
    with ZPublisher.WSGIPublisher.__old_load_app__(module_info) as ret:
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/", line 17, in __enter__
  File "/Users/pbauer/workspace/coredev/src/Zope/src/ZPublisher/", line 220, in load_app
    app = app_wrapper()
  File "/Users/pbauer/workspace/coredev/src/Zope/src/App/", line 78, in __call__
    return connection.root()[self._name]
  File "/Users/pbauer/workspace/coredev/bin/../lib/python2.7/", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'Application'

    4/4 (100.0%) test_regression (Products.CMFFormController.tests.testRedirectTo.TestRedirectToFunctional)

Error in test test_regression (Products.CMFFormController.tests.testRedirectTo.TestRedirectToFunctional)
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/", line 329, in run
  File "/Users/pbauer/workspace/coredev/src/Products.CMFFormController/Products/CMFFormController/tests/", line 71, in test_regression
  File "/Users/pbauer/workspace/coredev/src/Zope/src/Testing/ZopeTestCase/", line 43, in wrapped_func
    return func(*args, **kw)
  File "/Users/pbauer/workspace/coredev/src/Zope/src/Testing/ZopeTestCase/", line 127, in publish
    wsgi_result = publish(env, start_response)
  File "/Users/pbauer/workspace/coredev/src/Zope/src/ZPublisher/", line 254, in publish_module
    with load_app(module_info) as new_mod_info:
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/", line 17, in __enter__
  File "/Users/pbauer/workspace/coredev/src/Zope/src/Testing/ZopeTestCase/", line 73, in load_app
    with ZPublisher.WSGIPublisher.__old_load_app__(module_info) as ret:
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/", line 17, in __enter__
  File "/Users/pbauer/workspace/coredev/src/Zope/src/ZPublisher/", line 220, in load_app
    app = app_wrapper()
  File "/Users/pbauer/workspace/coredev/src/Zope/src/App/", line 78, in __call__
    return connection.root()[self._name]
  File "/Users/pbauer/workspace/coredev/bin/../lib/python2.7/", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'Application'

  Ran 4 tests with 0 failures, 2 errors, 0 skipped in 0.403 seconds.
Tearing down left over layers:
  Tear down in 0.000 seconds.
  Tear down in 0.010 seconds.
  Tear down in 0.068 seconds.
  Tear down plone.testing.z2.Startup in 0.007 seconds.
  Tear down plone.testing.zca.LayerCleanup in 0.001 seconds.

Tests with errors:
   test_attacker_redirect (Products.CMFFormController.tests.testRedirectTo.TestRedirectToFunctional)
   test_regression (Products.CMFFormController.tests.testRedirectTo.TestRedirectToFunctional)
Total: 10 tests, 0 failures, 2 errors, 0 skipped in 24.082 seconds.

24 seconds? I can work with that.

Still, I removed tests from each layer until I only had three tests left and reverted my changes to Products.Archetypes.

The result is the following:

./bin/test \
    --layer Products.Archetypes.tests.attestcase.Archetypes \
    --layer Testing.ZopeTestCase.layer.ZopeLite \
    --layer \
    -s Products.Archetypes \
    -s Products.CMFFormController \
    -t test_parseXML_empty \
    -t traversal_4981 \
    -t test_attacker_redirect \

Since more than one test still exposed the issue, I kept only very simple ones because I guessed that the issue is actually in the setup or teardown.

So next I changed the test test_parseXML_empty to a simple return. The error is still there. Trying the same with traversal_4981 makes it go away.

At this point I could skip reducing the layers since I only run three tests from two packages.

It was time to actually read what the remaining tests are doing. I stripped down all tests and their setup to the base minimum that still breaks the test run and could not find anything. I turn edCMFFormControllerImportConfiguratorTests into a ZopeTestCase and a PloneTestCase and realized that the error disappears when it is a PloneTestCase. Bad. Migrating the whole test to PloneTestCase or would be a lot of work since CMFFormControllerImportConfiguratorTests inherits from Products.GenericSetup.tests.common.BaseRegistryTests and does a lot of additional magic.

So the test layers for the two tests that did not fail or error by themselves but triggered the issue in the failing tests (traversal_4981 and test_parseXML_empty) seemed to be out of the scope of what I could do so I took a closer look at the failing tests themselves. I quickly found that I hate them but what they do is actually quite simple. Why do I hate them? Because they use the publish method of ZopeTestCase.Functional. That method (and its evil doctest-cousin Testing.ZopeTestCase.zopedoctest.functional.http) are way too clever helper methods that make things harder, not easier. I prefer to use restrictedTraverse or the testbrowser any time since both are much closer to what actually happens in the application.

This was the moment when I decided to migrate the tests in question to proper tests. It took me about 1 hour to create a pull-request which resolves the issue. The rest of the day was spent on a fruitless attempt to find the issue that must still be lurking somewhere between the three tests and their layers.

I hope that monster will never rear its ugly head again until CMFFormController is finally removed from the coredev. The PLIP 2092 by @esteele and me will remove the last remaining ControllerPageTemplates but there are some more left in Archetypes.

I fear it will be quite some time until all ZopeTestCase and PloneTestCase tests are migrated to The remaining happy thought is that many will not need to be migrated since they are part of Archetypes and will go awaaaaay with it.

Hello Thomas!

erstellt von Philip Bauer zuletzt geändert: 2017-03-02T15:56:48+01:00
We're excited to welcome Thomas Lotze to the team of

Thomas has many years of experience developing complex web projects with Python, especially using Pyramid and Plone. His interests also include automated testing and questions of software infrastructure. He shares our dedication to Open Source and we are looking forward to many exciting projects in the future.

Aside from development, Thomas does trainings on subjects that he knows well from day-to-day work, in particular Python, Pyramid and pytest.

You can meet us next at World Plone Day and PyConWEB in Munich.

World Plone Day - 28. April 2017 - München

erstellt von Steffen Lindner zuletzt geändert: 2017-02-17T21:12:59+01:00
Der World Plone Day ist eine weltweite Initiative der großartigen Community des Open Source Content-Management-Systems Plone.

Merkt euch den 28. April 2017 in euren Kalendern vor, an diesem Tag findet der World Plone Day in München statt. Die Location steht schon fest: Uni München, Oettingenstraße, Raum L 155.

Wir werden uns den halben Tag (14 bis 18 Uhr) intensiv mit Plone beschäftigen. Jeder der sich für CMS, Plone, Python und das Web interessiert, ist eingeladen. Die Teilnahme ist kostenlos. Am Programm feilen wir noch, möglich sind diese Themen: Plone 5.1, Quaive, PLOG und Shopsysteme mit Plone.

Details zum Programm sind hier zu finden.

Bei schönem Wetter geht es danach in den Biergarten am Chinesischer Turm!

Eine Anmeldung schalten wir bis Ende Feburar auf online.

Stay tuned!

Towards Plone 6

erstellt von Philip Bauer zuletzt geändert: 2017-03-01T11:52:13+01:00
A report from the Alpine City Sprint in Innsbruck

Two of the main items on the Plone roadmap are the move to Python 3 and a more active role in the development of the underlying stack of Plone, namely Zope. After the recent uptake of activity in development of Zope (see here and here) it was a sensible first task to try to get Plone to run on the most recent Zope stack. What exactly does that mean?

There is a PLIP that proposes to upgrade Plone's Zope-related dependencies and is approval by the Plone Framework Team to be a part of Plone 6. As with all PLIPs, the point is not only to agree on a task and a implementation but also actually doing the work. After two smaller efforts at sprints last year where only 2 or 3 people were working on the task we finally got a critical mass of developers together for the Alpine City Sprint. 15 people sprinted and worked on this one task.


Off to a good start

On the day before the sprint I rebased some of our branches and with a couple of quick and really dirty fixes I got Plone run on Zope 4. Many features were still not working but a working instance certainly provided motivation for this very dedicated sprint.

Bildschirmfoto 2017-02-13 um 11.43.41.png

WSGI replaces the ZServer

The ZServer is now a optional dependency and the new WSGI-Publisher is the default publisher making Plone fully WSGI-compliant! Thomas Schorr added some necessary buildout-parts to run Plone on uwsgi, waitress and gunicorn. Documentation is here and examples are here. Even the testbrowser used for all of Plone's functional tests now uses the WSGI-Publisher. The advantages of WSGI are obvious: It makes it easier to have interact Plone with other services and some features can be developed as WSGI-middleware. It'll be interesting to see if we will make more use of WSGI for Plone in the future than "simply" for serving.

ZCatalog 4 and ZODB 5

The result of the sprint is beyond what was originally proposed in the PLIP. We now not only use the master branch from Zope and the zope-toolkit packages (zope.interface, zope.component etc.) but also ZODB 5 and ZCatalog 4.

One change that required some work was that ZCatalog, as it now returns a empty list when you pass no query or a invalid query. Especially in the cases when you query the catalog for a non-existing index (portal_typos='foo') getting all brains from the Catalog seems like a bad idea. The down-side that change there is that there is no obvious way to walk through all brains in the catalog any more. Johannes Raggam added the convenience-method catalog_get_all for this.

The changes in ZODB are numerous but for Plone-developers nothing much really changes except that is supposed to be faster and supports Python >= 3.3 and 2.7. For the future it will be also interesting to discuss if switching to NewtDB would be a option. NewtDB runs ZODB 5 with RelStorage and stores data as json in Postgres.

Bildschirmfoto 2017-02-13 um 11.12.27.png

Python 3

The sprint did not try to move Plone to Python 3 yet but rather clear the way so we can do it. The last remaining blocker with C-code was RestrictedPython and Alexander Loechel and Michael Howitz continued working on the rewrite based on the Python AST module. At the end of the sprint they were happy to announce that it is no longer a blocker for our move to Python 3. All the other packages that rely on C-code that seems mysterious to mere Python-mortals (ExtensionClass, AccessControl, Acquisition, Persistence) were already ported to Python 3. RestrictedPython is a very useful tool that allows you to execute untrusted Python code that is loaded within a running Python program. It checks and modifies source code to only allow the execution of a restricted subset of Python.

What remains to be done?

  • jenkins_green.pngFix some failing tests (DONE): As of today we are down to 8 failing tests 6 failing tests 2 failing tests 0 failing tests!!! 
  • Update tasks and tickets
  • Merge changes: A lot of the changes were already merged in the master-branches for the current coredev (for Plone 5.1). But some merges will have to wait until Plone 5.1 is released because they only target Plone 6.0. Once all test are green and all our zope-related branches are merged we can make a release of Zope 4.0a3 and switch from using source-checkouts for everything (auto-checkout = *) to using released versions.
  • Documentation: We need to document what changed in Zope from the perspective of Plone. Most notably some lesser features of the ZMI are gone. These and other changes need to go into the upgrade-guide for the next Plone-Version. Also the new setup for various WSGI-Servers needs to be well-documented so that developers can  copy&paste best-practice configs into their buildouts. The documentation was started in a Google Doc where everyone can contribute. Once it's stable we'll move it to their respective places.
  • Upgrades: So far it seems like the changes we made did not make any upgrade-steps necessary. After adding a quick hack to work around the missing zope-controlpanel I was able to upgrade a existing Plone 5.0.6 site without any issues.
  • Profiling: Initial results with from plone.instanceprofiling indicate the speed of Plone has increased but we need more real-life tests before a final judgement. It will also be interesting to see the impact of Python 3 on speed.

Bildschirmfoto 2017-02-13 um 11.15.54.png

Inside the sprint

As usual during the sprint we ran into a bunch of blockers and hard nuts to crack. The hardest one was probably that the current users roles were mysteriously dropped during a request after running trough some C-code. Like the memory of a bad dream, the bug vanished in the light of the morning - actually someone switched to different branch of AccessControl. On the first day already, Matthew Wilkes had committed to a metaclass in Persistence (thankfully brown-bagging my terrible late-night hack and replacing it with something much more elegant), and installed gdb to debug the issue with the dropped roles.

Unfortunately many of our tests were broken since the testlayer did not use chameleon, thankfully after a lot of debugging the solution found by Thomas Schorr was pretty simple. Also tough was a issue that broke tests in weird ways. Gladly Jens Klein, our host, dug deep and found the culprit: Combining bundles modified the response's Content-Type header, surprise! A issue similar to

Bildschirmfoto 2017-02-13 um 11.22.11.png


By the way: We also had a demo of and a discussion about shop-solutions for Plone. has plenty of great features and seems really well thought-through. It still lacks some end-user documentation with screenshots and a new release but we will certainly use it in our next project that involves a shop.

Where do we go from here?

To the pool in Sorrento! Plone Open Garden (PLOG) in beautiful Sorrento is the perfect opportunity to discussing the further roadmap to Python 3 and continue working on this. PLOG will have a strong focus on the headless CMS project for which a Python 3-compatible Plone 6 will be a mayor milestone. Register now and enjoy a week at the pool with the Plone community.

2016-03-26 18.03.01.jpg

And then on to Finnland to the Midsummer Sprint! All the work on the backend needs to be balanced with some fun UX and UI work to enhance the content editors experience.


We worked on many important parts of the Plone Roadmap: Removing unused components, clean up the code base, moving to Python 3, and making Plone WSGI compliant. There is now a good chance that Plone will be on Python 3 this or next year. A great sprint, dedicated and smart people, good beer and food and nice mountains. Thanks to the organizers of Klein und Partner who worked hard to keep us fed, fueled and entertained and thanks to all the sprinters who dedicated so much of their time for such a great open source project!

Update (19.02.2017)

  1. Last week gocept announced there will be another Zope sprint in Mai with the goal to port Zope to Python 3. Already 11 developers signed up.
  2. David Glick fixed the remaining failing tests (mosty issues with the new handling of exceptions).


Push coverage report out of an Gitlab CI Runner

erstellt von Steffen Lindner zuletzt geändert: 2017-02-06T17:26:17+01:00
For our Plone/Python projects we often generate coverage reports as HTML sites, this posts show how you can push this report out of the Gitlab CI Runner.

 Save FTP Password & Login as secret variables in Gitlab > Settings > Variables. They can be accessed in the .gitlab-ci.yml as $FTPLOGIN & $FTPPASSWORD.


Here is our .coveragerc file, which sets the output directory:

include =
omit =
directory = parts/test/htmlreport
title = Plone Project

And our .gitlab-ci.yml:

    - /usr/bin/virtualenv .
    - ./
     - export DISPLAY=:99
     - ./bin/code-analysis
     - ./bin/coverage erase
     - ./bin/coverage run -p --source=src bin/test || exit 1
     - ./bin/coverage combine
     - ./bin/coverage html
     - ./bin/coverage report
        # Stage robot
     - ./bin/test --all -t 'robot' || exit 1
     - cd parts/test/htmlreport/ && find . -type f -exec curl --ftp-create-dirs -T {} -u $FTPLOGIN:$FTPPASSWORD$CI_BUILD_ID/{} \;
        expire_in: 1 week
            - parts/test/

The interesting part is the curl one, it uploads all coverage files to a FTP server.

If your static webserver is public accessible use basic auth with a .htaccess file:

htpasswd -c .htpasswd starzel

.htaccess file:

AuthUserFile /var/www/htdocs/.htpasswd
AuthGroupFile /dev/null
AuthName "Please Enter Password"
AuthType Basic
Require valid-user

Better integration into Gitlab coming soon as Gitlab Pages see this Pull Request.


  - Cron to delete old directories.

  - Upload files if robot / code-analysis failed.


Fix failing parallel running browser tests with Gitlab & Robotframework

erstellt von Steffen Lindner zuletzt geändert: 2017-01-25T16:41:19+01:00
One of our project is organized in sprints where all developer work on the same code at the same time. We use one Gitlab CI server with a simple shell executer and had random failing builds with Robotframwork & Firefox.

Our Gitlab CI Runner can run three shell executer at the same time. Sometimes two Robotframework (Selenium) tests run in parallel and one of them fails with this error:

error: [Errno 98] Address already in use

That means Firefox is already in use (by the other build) and the CI job failed and blocked our process. Gitlabs automerge -feature does not happen and the code-reviewer needs to manually start the job again since it is marked as a fail.

We solved the problem with a little shell script that waits if a Robot process is running:


while :

RESULT=`pgrep -f robot`

if [ "${RESULT:-null}" = null ]; then
echo "Robot not running, starting "
./bin/test --all -t robot || exit 1
echo "running"
sleep 20

Our .gitlab-ci.yml file looks like this:

- export PATH=$PATH:bin/
- export DISPLAY=:99

# stage: build
# Stage: build
- /usr/bin/virtualenv .
- ./
# Stage test
- ./bin/code-analysis
- ./bin/i18nize_all && ./bin/podiff_all
- ./bin/coverage erase
- ./bin/coverage run -p --source=src bin/test || exit 1
- ./bin/coverage combine
- ./bin/coverage html
- ./bin/coverage report
# Stage robot
- ./bin/ || exit 1
expire_in: 1 week
- parts/test/

The call to start the robot tests (./bin/test --all -t robot) is wrapped into the ./bin/ script.

If you are a buildout user, this might be useful to get the script integrated:

parts +=

recipe = collective.recipe.cmd
on_install = true
on_update = true
cmds = cp ${buildout:directory}/templates/ ${buildout:bin-directory}/

 Hope this helps someone!

Plone 5 Release Party and the Plone Theming Sprint

erstellt von Philip Bauer zuletzt geändert: 2015-09-12T13:03:42+01:00
Let's celebrate the release of Plone 5! On September 16 there is a release party taking place at Kitchen2Soul, Schlörstraße 4, Munich.

Eric Steele, our beloved release manager, is going to highlight some of the main features. Afterwards there is food, drink and fun.

The party is also the kick-off to the Plone 5 Theming Sprint in Munich on 16 - 20 September 2015

The main task of the sprint is to document and improve the theming-story for Plone 5. So...

  • if you want to learn about theming and customization with Plone
  • if you do theming and customization with Plone
  • if you want to upgrade a existing theme to Plone 5
  • if you want others to have a great theming and customization-experience with Plone

...please consider coming to the sprint. Everybody is welcome!

During the sprint we are going to have several talks related to theming:

  • Wednesday at 1 p.m.: An introduction to Plone 5 Theming. By Víctor Fernández de Alba
  • Thursday at 6 p.m.: The Plone intranet initiative and its design first. By Philip Bauer and Alexander Pilz
  • Friday at 6 p.m.: Patternslib and Mockup. The javascript libraries that bring webdesign and development together. By J.C. Brand

You can find all the details here:

Details about the Party:

  • 6 p.m. in Kitchen2Soul, Schlörstraße 4
  • 7 p.m. presetation of Plone 5 by the Release Manager - Eric Steele
  • Afterwards Party - Open End
  • Please let us know if you are coming using the form at

Test Plone 5 now:
Learn more about Plone:

Ansible DebOps and how to move Gitlab to it

erstellt von Steffen Lindner zuletzt geändert: 2015-03-24T14:09:17+01:00
This howto shows how to move a non-debops Gitlab installation to a debops-managed installation (and let debops do upgrades).

For some time we manage our infrastructure with Ansible, mainly we plugged together several roles from the Ansible Galaxy (the pypi of Ansible roles). It helped us get our servers into a better state without that much effort. our ownCloud Hosting benefits a lot from this move.

In search for a Jenkins role, we discovered Debops. DebOps is "a collection of Ansible playbooks, scalable from one container to an entire data center." It does not have a Jenkins role yet (we are working on it), but it really shines in Debian/Ansible best-practices and well developed and integrated Playbooks. Maciej Delmanowski the inital creator of DebOps is outstanding helpful and constantly pushing new features and roles.

Recently we transfered two Gitlab installations (and updated it). Here is how we did it:

You need a working debops gitlab install on a new server. See Getting started and debops.gitlab.

To be able to import a backup both Gitlab Installations need be the same version (and same database):

On the old server:

$ su git
$ cd gitlab
$ git rev-parse --verify HEAD
$ 1660aa23e3f6bea8e0de54a420e29953f6bd194f #Save the hash
$ cd gitlab-shell
$ git rev-parse --verify HEAD
$ a3b54457b1cd188981d4d0775fc7acf2fd6aa128 #Save the hash
# Do the backup
$ bundle exec rake gitlab:backup:create RAILS_ENV=production
# Omnibus install
$ gitlab-rake gitlab:backup:create

Transfer the backup from gitlab/tmp/backups/1423229329_gitlab_backup.tar to the new server /var/backups/gitlab.

On the new server:

$ su git
$ cd gitlab
$ GIT_WORK_TREE=/var/local/git/gitlab git checkout -f 1660aa23e3f6bea8e0de54a420e29953f6bd194f
$ cd gitlab-shell
$ GIT_WORK_TREE=/var/local/git/gitlab-shell git checkout -f a3b54457b1cd188981d4d0775fc7acf2fd6aa128
$ cd gitlab
$ mysql # Login into mysql shell
# Drop the fresh gitlab db from newly installed gitlab
$ drop database gitlabhq_production;
$ bundle install
$ bundle exec rake gitlab:backup:restore RAILS_ENV=production BACKUP=1423229329 #With backup timestamp
# Omnibus install
# gitlab-rake gitlab:backup:restore
$ /etc/init.d/gitlab restart
# Check if gitlab is working and your data is there
$ /etc/init.d/gitlab stop
$ cd src/
$ GIT_WORK_TREE=/var/local/git/gitlab-shell git checkout master
$ cd src/
$ GIT_WORK_TREE=/var/local/git/gitlab git checkout master
# Run debops against the host to let it upgrade
$ debops -l <hostname> -t gitlab

Lean back and let DebOps help you with coming upgrades of Gitlab.

Useful links:

Magic templates in Plone 5

erstellt von Philip Bauer zuletzt geändert: 2017-03-01T12:22:05+01:00
Due to the new rendering-engine chameleon it is fun again to write templates

Plone 5 uses Chameleon a its rendering engine. Did you know that because of that you can put a pdb in a template? If you saw the keynote by Eric Steele on Plone 5 you probably do.

But did you also know that the variable econtext holds all current variables up to the moment the pdb is thrown?

Let's put a pdb in


<metal:content-core fill-slot="content-core">
    <metal:block define-macro="content-core"
          tal:define="templateId template/getId;
                      scale_func context/@@images;
                      scaled_image python: getattr(context.aq_explicit, 'image', False) and scale_func.scale('image', scale='mini')">

<?python import pdb; pdb.set_trace() ?>

    <figure class="newsImageContainer"
         tal:condition="python: scaled_image">
        <a href="#"
           tal:define="here_url context/@@plone_context_state/object_url;
                       large_image python: scale_func.scale('image', scale='large');"
           tal:attributes="href large_image/url">
          <img tal:replace="structure python: scaled_image.tag(css_class='newsImage')" />


When rendering a News Item the variable scaled_image is accessible as econtext['scaled_image']:

> /Users/philip/workspace/test/
(Pdb) econtext['scaled_image']
<plone.namedfile.scaling.ImageScale object at 0x110073b90>
(Pdb) econtext['scaled_image'].width

You can inspect the whole econtext:

(Pdb) from pprint import pprint as pp
(Pdb) pp econtext
{'__convert': <function translate at 0x10fb4d7d0>,
 '__decode': <function decode at 0x10fb4d578>,
 '__slot_content_core': deque([]),
 '__slot_javascript_head_slot': deque([]),
 '__translate': <function translate at 0x10fb4d7d0>,
 'ajax_include_head': None,
 'ajax_load': False,
 'args': (),
 'body_class': 'template-newsitem_view portaltype-news-item site-Plone section-super-news userrole-manager userrole-authenticated userrole-owner plone-toolbar-left-default',
 'checkPermission': <bound method MembershipTool.checkPermission of <MembershipTool at /Plone/portal_membership used for /Plone/super-news>>,
 'container': <NewsItem at /Plone/super-news>,
 'context': <NewsItem at /Plone/super-news>,
 'context_state': <Products.Five.metaclass.ContextState object at 0x10db00910>,
 'default': <object object at 0x100291bf0>,
 'dummy': None,
 'here': <NewsItem at /Plone/super-news>,
 'isRTL': False,
 'lang': 'de',
 'loop': {},
 'modules': <Products.PageTemplates.ZRPythonExpr._SecureModuleImporter instance at 0x102e31b48>,
 'nothing': None,
 'options': {},
 'plone_layout': <Products.Five.metaclass.LayoutPolicy object at 0x10db00310>,
 'plone_view': <Products.Five.metaclass.Plone object at 0x10db00dd0>,
 'portal_state': <Products.Five.metaclass.PortalState object at 0x10fbe8d50>,
 'portal_url': 'http://localhost:8080/Plone',
 'repeat': {},
 'request': <HTTPRequest, URL=http://localhost:8080/Plone/super-news/newsitem_view>,
 'root': <Application at >,
 'scale_func': <Products.Five.metaclass.ImageScaling object at 0x10c2bf390>,
 'scaled_image': <plone.namedfile.scaling.ImageScale object at 0x110073b90>,
 'site_properties': <SimpleItemWithProperties at /Plone/portal_properties/site_properties>,
 'sl': False,
 'sr': False,
 'target_language': None,
 'template': <Products.Five.browser.pagetemplatefile.ViewPageTemplateFile object at 0x10fff3ed0>,
 'templateId': '',
 'toolbar_class': 'pat-toolbar initialized plone-toolbar-left',
 'translate': <function translate at 0x10fb4d7d0>,
 'traverse_subpath': [],
 'user': <PropertiedUser 'adminstarzel'>,
 'view': <Products.Five.metaclass.SimpleViewClass from /Users/philip/workspace/test/src-mrd/ object at 0x10cd93910>,
 'views': <Products.Five.browser.pagetemplatefile.ViewMapper object at 0x10f5cc190>,
 'wrapped_repeat': <Products.PageTemplates.Expressions.SafeMapping object at 0x10ffba1b0>}

Using n you can actually walk down the template and inspect new variables as they appear. After pressing n about 11 times the variable large_image appears as econtext['large_image'].

(Pdb) econtext['large_image'].width

The pdb-session you are in is no restricted python but real python. This means you can do the following:

(Pdb) from plone import api
(Pdb) portal = api.portal.get_tool('portal_memberdata')
(Pdb) memberdata = api.portal.get_tool('portal_memberdata')
(Pdb) memberdata.getProperty('wysiwyg_editor')

Hey, what is kupu doing there? I found that in some of our sites that were migrated from Plone 3 this old setting prevented TinyMCE to work in static portlets. But that is a different story, let's just get rid of it.

(Pdb) memberdata.wysiwyg_editor = 'TinyMCE'
(Pdb) import transaction; transaction.commit()

This is a full-grown pdb and you can inspect and modify your complete site with it.

But there is more: You can actually have complete code-blocks into templates:


from plone import api
catalog = api.portal.get_tool('portal_catalog')
results = []
for brain in catalog(portal_type='Folder'):


    <li tal:repeat="result results">

Quick and dirty? Maybe dirty but really quick! It is still very true that having logic in templates is bad practice but I think in some use-cases it is ok:

  • Debugging
  • When you customize a existing template (with z3c.jbot or and need some more logic
  • When you quickly need to add some logic to a browser-view that only has a template but no class of it's own

Have fun templating with Plone 5! If you want to learn more about Plone 5 you can still register for the training "Mastering Plone 5 Development" in March 2.-6. (


As disucced here you can add the econtext to the the locals() by using the following stanza:

<?python locals().update(econtext); import pdb; pdb.set_trace() ?>


Plone Conference 2014: The Highlights

erstellt von Philip Bauer zuletzt geändert: 2017-03-01T12:22:06+01:00
The Plone Conference has once again proven its value. There were many excellent talks and everyone had a great time. In Open Spaces and during many discussions between talks the current state and the future of Plone became much clearer.

Plone 5

The main thing is Plone 5. It look great, is a huge step forward in terms of user experience and brings tons of important improvements. The keynote of Plone's release manager Eric Steele showcases all these things.

Getting Plone 5 ready for a beta-release is the top item on everyone’s agenda. That effort not only involves finishing projects such as mockup, the new theme and contenttypes but also writing documentation for end-users, the upgrade-guide for developers and testing.

The Plone Roadmap

The second biggest thing was the roadmap-discussion. Discussions about the a new Roadmap started in October 2013 at the Plone Conference in Brazil under the catchy title "Plone Roadmap 2020". The initial topic was the future of Zope within Plone's ecosystem, but it turned into a broader discussion about all the things that the community wants to change in Plone, not forgetting the whys and hows.

Roadmap progress in Bristol 2014

First we looked at reasons why we use Plone. The following motivations were mentioned the most:

  • The Plone Community
  • Fully-featured user friendly product that is directly usable
  • Flexibility of the software architecture
  • It creates jobs
  • Best security-record
  • Open source

As Martin Aspelli put it so eloquently: If we are going to change Plone, can we please not mess any of the above up?

Next we compiled a list of things that we want to change in Plone in the next five years.

  • python-api
  • json-api
  • Improve end user experience
  • Documentation and training: Document the recommended way to do things
  • Improve TTW-story in theming, templating and customization
  • Simplify code base, reduce number of eggs, remove legacy technologies
  • Remove dependency on CMF
  • Roadmap communication

Not a single person advocated a rewrite since it is clear that a rewrite would mess with some things on the first list.

When looking closer at the aims it becomes clear that all of these things are already being worked on:

  • python-api: There is now a Plone Improvement Proposal to ship plone.api with Plone and also use it in the core. There will be further discussions about what we need from an api to further isolate Plone from parts that we might want to replace in the future.
  • json-api: The community is already working on a RESTful json-api for Plone that will allow isolating the front-end from the backend and experiment with various javascript-frameworks. With this api Plone will be better suited for for web-applications where Plone is "only" used as the backend and will be a friendly citizen in a world of mixed technologies.
  • UX: Plone 5 will already be a huge step in the right direction but there will always be room for improvement. The enthusiastic reaction to the efforts of the Plone Intranet Consortium showed how important a good UX is.
  • Documentation: The new are much better than anything Plone had in the past. During the sprint the documentation was already being cleaned of examples that use grok. Also the documentation for the Mastering Plone Training teaches always the recommended ways to do things and will be expanded and upgraded for Plone 5.
  • TTW-Story: Although Mosaic (the new name for Deco) looks extremely promising this is surely one of the areas where Plone still needs to invest more. For template-customization we still rely on old technologies: The only way to customize a viewlet within Plone is to use the ZMI (not recommended!). The community will have to agree on achievable solutions to have some real progress. But do not forget that Plone already has a great TTW-story: Dexterity and the Diazo theme-editor are powerful features and blow the competition out of the water.
  • Simplify code base, reduce number of eggs: Many technologies (formlib, portal_skins, plone-tools, cpy/cpt) are already deprecated and the code is migrated to browserviews and z3c.form (*cough*) in Plone 5. There is also a PLIP that aims to move many into the core-package Products.CMFPlone [].
  • Remove dependency on CMF: Without a python-api we cannot remove parts of CMFCore and Zope. Having that api will give us the options we need to replace/remove stuff. We'll need some experiments and even more discussions about what we want to remove and what to replace it with.
  • Roadmap communication: The discussion on a the Plone Roadmap will be continued. The next Plone Open Garden (April 2015 in Sorrento) will probably be turned into a Plone Strategic Planning Summit. Communicating the community's strategic vision of Plone in the short, mid and long term to the public is almost as important as agreeing on a roadmap and implementing it.

Since there is no official roadmap-team these kind of meetings are where the future of Plone is actually agreed on. This is where Plone happens and everyone who takes part is part of the roadmap-team.

The Plone Intranet Consortium

One of the most anticipated talks of the conference was about the Plone Intranet Consortium. Since its foundation last year, the consortium (of which is a founding member) is working on a competitive Social Intranet solution. One of the aims is to evolve and strengthen the position of Plone in a commoditized market. The design-first approach and the fact that Plone-companies put their resources together allows for high expectations. See the talk:


The sprint

As usual after the conference there were two days to actually work on Plone. The hotel had to quickly open another big room to accommodate all sprinters since many more people showed up this time than  expected. On saturday morning you could see the titanpad exploding when people started adding the topics they wanted to work on. I worked with a great team of developers on the default contenttypes of Plone 5, mainly focusing on making it easy to migrate the content of existing websites to Dexterity. Other results can be seen at the titanpad and here.

In 2015 the Plone Conference will be in Bucharest. I wouldn't miss it for the world.


Extended Mastering Plone Training in Munich in Spring 2015

erstellt von pgerken — zuletzt geändert: 2014-11-11T19:16:14+01:00
We love to teach more than what is possible in two or three days. Because of this, we will offer an extended Plone training in wonderful Munich in spring 2015

When we prepare a training, we always create more material than what we can walk through in the short amount of time we have available.

We decided that we want to offer you more by extending the training period to 5 days. In 5 days we can cover the material deeper have more exercises and additional topics.

The additional topics include:

  • How to work with relations
  • How to test everything using and robot-framework
  • Building forms and control-panels with z3c.form

We would love to have you with us.

More information is available on our dedicated training page.

Update: By popular demand we moved this training from november 2014 to 2. - 6. March 2015. We will  inform everyone who contacted us about the training about the new date.  

A reminder about catalog-indexes

erstellt von Philip Bauer zuletzt geändert: 2014-05-21T05:56:13+01:00
We often create new indexes to make content searchable in Plone. Many developers still do it wrong.

Most new sites we create contain facetted search based on eea.facetednavigation. Thus we often create new indexes to make content searchable. One example is where attributes and relations of projects are used to make them searchable in a useful way.

Almost everything that is to be said about indexes can be found in

But there was a known pitfall when registering indexes in catalog.xml that was only fixed in Plone 4.3.2. Even though Maurits van Rees warned about this in a blogpost which is also reference in the documentation I often see developers making that mistake.

Unless working with Plone 4.3.2+, you should never register a index in catalog.xml because the index will be purged when reinstalling your package. Instead register new indexes in your This was fixed in GenericSetup 1.7.4.

I firmly believe that addons have to be reinstallable without ruining a site.

I extended the method written by Maurits to separate the name of the index and the indexed method (useful when dealing with old code):

# -*- coding: UTF-8 -*-
import logging
from Products.CMFCore.utils import getToolByName
PROFILE_ID = 'profile-my.package:default'

def setupVarious(context):

    # Ordinarily, GenericSetup handlers check for the existence of XML files.
    # Here, we are not parsing an XML file, but we use this text file as a
    # flag to check that we actually meant for this import step to be run.
    # The file is found in profiles/default.

    if context.readDataFile('bgp.webcode_various.txt') is None:

    # Add additional setup code here
    logger = context.getLogger('my.package')
    site = context.getSite()
    add_catalog_indexes(site, logger)

def add_catalog_indexes(context, logger=None):
    """Method to add our wanted indexes to the portal_catalog.


    When called from the import_various method below, 'context' is
    the plone site and 'logger' is the portal_setup logger.  But
    this method can also be used as upgrade step, in which case
    'context' will be portal_setup and 'logger' will be None.
    if logger is None:
        # Called as upgrade step: define our own logger.
        logger = logging.getLogger('my.package')

    # Run the catalog.xml step as that may have defined new metadata
    # columns.  We could instead add <depends name="catalog"/> to
    # the registration of our import step in zcml, but doing it in
    # code makes this method usable as upgrade step as well.  Note that
    # this silently does nothing when there is no catalog.xml, so it
    # is quite safe.
    setup = getToolByName(context, 'portal_setup')
    setup.runImportStepFromProfile(PROFILE_ID, 'catalog')

    catalog = getToolByName(context, 'portal_catalog')
    indexes = catalog.indexes()
    # Specify the indexes you want, with
    # ('index_name', 'index_type', 'indexed_attribute')
    wanted = (('myindex', 'FieldIndex', 'getMyAttribute'),
    indexables = []
    for name, meta_type, attribute in wanted:
        if name not in indexes:
            if attribute:
                extra = {'indexed_attrs': attribute}
                catalog.addIndex(name, meta_type, extra=extra)
                catalog.addIndex(name, meta_type)
            if not attribute:
                attribute = name
  "Added %s '%s' for attribute '%s'.", meta_type, name, extra)
    if len(indexables) > 0:"Indexing new indexes %s.", ', '.join(indexables))

By the way: Besides many other amazing features the package ftw.upgrade also has methods catalog_rebuild_index, catalog_add_index and catalog_remove_index you an use in your upgrade-steps.

Mastering Plone

erstellt von Philip Bauer zuletzt geändert: 2014-05-20T16:47:00+01:00
A three-day introduction into customizing and developing websites with Plone

tl;dr: We're giving our three-day "Mastering Plone" training in Munich (in english)

During the course of this three-day training we will teach how to

  • ... wield the awesome features of Plone for maximum effect.
  • ... customize and extend Plone to make it do exactly what you want.
  • ... use the current best-practices to become a Plone rockstar.

In the first part we'll teach the fundamentals needed to setup and manage your own website using the many build-in features of Plone.

The second part will focus on customizations that may be done through-the-web.

The third and longest part will be devoted to Plone-development. Here we focus on current best-practices like Dexterity and Diazo that make developing with Plone fun and accessible.

The training is for people who are new to Plone or used to work with older versions of Plone and want to learn the current best-practices. Basic Python and HTML/CSS skills are a requirement.

The course is an updated and expanded version of the trainings we held at the international Plone-Conferences in Arnhem and Brasilia. The documentation for these can be found at

As always, we give the training as a team of two Trainers. This way you'll receive a 1on1 training as soon as something works differently than expected. Something that is not possible with a single trainer and something that adds a lot of insight when something did not work as expected.

If you're interested call us at +49 (0)89 - 189 29 533 or send a mail to

26. - 28. May 2014

9:00 - 18:00

Schwanthalerstr. 80
80336 München

Philip Bauer
Patrick Gerken


EUR 1000.- per person plus 19% MwSt (VAT)
Please talk to us if you might be eligible to a discount (Student, Member of the international Plone-community and so on...)


Update (20.05.2014):
Due to illnesses we currently have three open slots for the training. Talk to us if you are interested.

We are working hard on updating the training. Here are some highlights:

  • We now use from the beginning
  • More dexterity-types (without grok)
  • Custom search using eea.facetednavigation
  • Infos about debugging
  • We now teach the use of plone.api
  • ...

How to embed self signed certs and how to avoid not verifying https URLs with git

erstellt von pgerken — zuletzt geändert: 2014-04-11T14:07:33+01:00
Working with self signed certificates can be hard. Here is how you can avoid forgoing certificate validation in git with self signed certificates.

One of our Clients has high demands on the protection of its data and code.

They run their own, self hosted git server and it is only accessible via https.

The original suggestion to make it work, was to deactivate ssl certificate validation on our computers. Luckily, git allows for a simpler solution, even though it's not straight forward to get there.

First, you need the public key of the certification authority that signed the the ssl key the server uses. A self signed certificate signed is its own certification authority. The proper way would be to ask the client to provide the certificate via a secure channel. The improper way is to just download the certificate. Improper, because how do you know it is the right certificate. It is still better than not doing certificate validation at all.

Getting the certificate the insecure way

You can ask a webserver to return its certificate and CA certificates with openssl:

openssl s_client -showcerts -connect yourserver:443 </dev/null

This returns a lot of data you don't want all of it, only the last block that starts with "BEGIN CERTIFCATE" and ends with "END CERTIFICATE". Here is the same command with a pipe to a little awk script to do just that:

openssl s_client -showcerts -connect yourserver:443 </dev/null | awk '/-----BEGIN CERTIFICATE-----/ {start=1; cert=""};/-----END CERTIFICATE-----/ {start=0; cert=cert $0; };{if (start) cert=cert $0 "\n"}; END {print cert}' > yourserver.pem

Now that we have the certificate, we must teach git to use it.

We can teach git to use the certificate, but if we use it globally, we would not be using the standard CA certificates any longer. So we can only use the certificate for specific repositories.

While we can easily modify the configuration for a specific git checkout, we must first check it out.

How to use the new certificate with git

To bootstrap git, you enter:

GIT_SSL_CAINFO=yourserver.pem git clone https://YOURSERVER.../

Now, we can add the specific entry to the configuration in PROJECT/.git/config at the end of the file

    sslCAInfo = /home/mememe/project/yourserver.pem

Please be aware that you must enter the full path.

Now you are done and can securely use git pull and push.

How to use it with mr.developer

Bootstrapping a buildout with mr.developer can be done with the same prefix as the git clone. Unfortunately it will NOT work if you have mixed git repositories and need more certificates, for example for github https urls. You could avoid it by downloading the github certificate too, and attach it to the already existing pem file.

Sprinting towards Plone 5

erstellt von Philip Bauer zuletzt geändert: 2017-03-01T12:22:06+01:00
The drive towards Plone 5 is gaining momentum as the goal seems to be in a reachable distance.

For the first time there actually is a state of Plone 5 that is can be shown to people who are interested in the future of Plone. The essential building-blocks of Plone 5 are starting to look presentable: The new barceloneta-theme is really nice and comes with the new toolbar and the new widgets.

Don't get me wrong, there is still quite some work to be done before a final release but the first alpha-release is in the making and the current state can actually be shown. These screenshots are from a fresh build of the coredev.buildout.

plone5-left-2.png plone5-top-2.png plone5-edit-2.png

To reach Plone 5 the Plone-community is organizing a staggering number of sprints. My sprint-plan for 2014 looks like this: Cologne, Munich, Munich (again), Sorrento, Berlin, Bristol.

I recently attended the cathedral-sprint in Cologne, organized by the tireless Timo Stollenwerk and generously sponsored by GFU Cyrus AG. Timo, whose sole purpose in life sometimes seems to be to keep Jenkins happy, organized another impressive display of the spirit of the Plone-community as 30 people from around the world gathered there to work in groups on pushing Plone 5 towards readiness. It is great to see so many originally separate efforts coming together smoothly and making a whole new thing that will be at the core of Plone 5.

Together with Johannes Raggam, Wolfgang Thomas and some help from others I was continuing the work on and we made good progress. The current version 1.1b2 of the new default-content-types for Plone 5 has a multitute of bugfixes and some relevant changes compared to version 1.0:

  • Collections are now a behavior that can be enabled on any content-type. The collection-type is basically a folder with that behavior.
  • The shiny new event-type now uses the behaviors provided by including support for recurring events and timezones.
  • Richtext-fields are now a behavior shared by most types.
  • We now have migrations from 14 different content types to the new types including migrations for four (!) iterations of the event-type.

I have been using since before the project officially existed and I think every plone-developer should give it a try and report any issues especially regarding migrations to make it safe to ship with Plone 5.

It was a real pleasure being reunited with old friends and making new ones. It is especially encouraging that even at sprints there are always new people showing up and contributing to Plone.

The next big sprint on my agenda is the Wine-and-Beer-Sprint in Munich. For the second time we are organizing this sprint together with our friends from and again it is a double-sprint with a leg in Munich and one in Capetown. We will continue working towards Plone 5. The necessary tasks to get Plone 5 out of the door are summarized in this ticket:

Register now since we only have a limited amount of places! If you really need additional incentive to attend, here they come: A fine selection of Single Malt Whiskeys, a great sprint-location that is accessible 24h, Sichuan Hot Pot and white sausages (not to be mixed).

The Wine and Beer-sprint is is preceded by the DocSprint (also in Munich) that focuses on consolidating, improving and versioning(!) the documentation for Plone.

See you in Munich and elsewhere!

How to handle unknown meta_type in GS

erstellt von pgerken — zuletzt geändert: 2013-05-15T09:21:27+01:00
We were lost when supporting multiple Plone versions with generic setup and the typeinfo step. Soon there won't be the Topic type any more. Removing declarations for the Topic type and leaving users of old Plone-Versions behind is no option. We found a way to handle it in a single profile.

Our package collective.noticeboard should work with somewhat older Plone sites and new and shiny ones that ditch AT in favor of using DX all the way down.

Therefore we must declare an available view for Topics even though Topics might not exist in the targeted Plone sites.

Trying to import our profile in such a case results in a traceback like that:

Module Products.GenericSetup.utils, line 509, in _importBody
Module Products.CMFCore.exportimport.typeinfo, line 210, in _importNode
Module Products.GenericSetup.utils, line 570, in _initObjects
- __traceback_info__: ('Topic', '')
ValueError: unknown meta_type ''

We don't want to create confusion by adding multiple profiles and we don't want to enable different profiles in zcml based on import statements, because if you have a Topic or not is decided during runtime.

Fortunately there is an undocumented feature hidden in GS: You can declare your type declaration as deprecated. This way, GS does not try to initialize the object and does not try to create the type information object if it does not exist.

Be careful about this directive, this works for the typeinfo step, you have to test, if it does not miss important actions when used in other steps.

This is our new types.xml:

<?xml version="1.0"?>
<object name="portal_types"
        meta_type="Plone Types Tool">

    <object name="Folder"/>
    <object name="Topic" deprecated="True"/>
    <object name="Collection"/>
    <object name="Plone Site"/>


World Plone Day 2013 in Munich

erstellt von Steffen Lindner zuletzt geändert: 2013-04-12T13:07:48+01:00
On April 24 another WPD is upon us. We'll meet in the Seidlvilla in the heart of Munich-Schwabing.

On April 24 another WPD is upon us. We'll meet in the Seidlvilla in the heart of Schwabing.

This year will be presenting three talks (in german):

  • "What is Plone", a presentation of features, use-cases and the roadmap by Philip Bauer.
  • "The Plone eco-system", a wild ride through the architechture underlying Plone and how it relates to other technologies by Steffen Lindner and Patrick Gerken
  • "Pimp my Plone", a very biased presentation of favorite addons by Philip Bauer

The complete Programm can be found here. There is plenty of time for questions and discussions that can be continued into the evening when we move to a beergarden.

Attendance is free, there will be food, but sadly no cupcakes (picture taken from

Back from the Plone Open Garden in Sorrento

erstellt von Philip Bauer zuletzt geändert: 2013-04-09T19:08:30+01:00
If you do Plone, come to PLOG next year. Bring your family.

A poster I once designed for World Plone Day said "There is no place like Plone".


The community surrounding Plone is in fact like family for many of us. The Plone Open Garden in Sorrento is one of the events where the Plone-family meets the Plonistas family. It was a near-perfect mixture between a holiday, a sprint and a conference.

Imaging sitting under palmtrees drinking a nice cappucino and doing some light coding while the former president of the Plone foundations splashed around in the pool with his daughter. Add to that the sight of Vesuvius, loads of italian food, great presentations and many people to have engaged discussions with. Bliss.


You can even go hiking in the hills above Sorrento

It was also a very productive week. We managed to get a lot of work done  in that should make it ready for Plone 4.4. I gave a short talk about its state. The slides are not too informative without the demo. But I encourage everyone to use it for your next project or even try migrating a website. A new release will be out in a couple of days.

Thanks to the guys from Abstract for organizing PLOG, special thanks go to Maurizio for his infectious enthusiasm and optimism.

In short: If you do Plone, come to PLOG next year. Bring your family.

See you next year. Be prepared to meet my family :-)

wpd.countdown portlet - Counting days to World Plone Day

erstellt von Steffen Lindner zuletzt geändert: 2013-02-22T19:58:00+01:00
wpd.countdown is a portlet for promoting the upcoming World Plone Day 2013.

To support the upcoming World Plone Day, we created wpd.countdown, a portlet counting down the days to the World Plone Day - 24.4.2013.

Help us spread the word about the World Plone Day and embed the portlet into your Plone site. The installation is easy and documented on PyPi. The portlet has a nice configlet for customizing the date and the linked URL. If you host an WPD Event, put your link in.

The code is in the collective on Github. We already added the Portlet on and :)

wpd.countdown was developed by based on wpd.mmxi.countdown by Simples Consultoria.

collective.noticeboard - a noticeboard for Plone

erstellt von Philip Bauer zuletzt geändert: 2013-02-05T16:10:11+01:00
collective.noticeboard is a Plone add-on for fancy virtual noticeboards. The board handles as intuitively like its real-live cousin.

Once you install collective.noticeboard, any folder or collection (even the site itself) can be viewed as a noticeboard. It displays the content of that folder/collection as colored sticky notes that can be moved, resized and edited.

collective.noticeboard comes with a per-board-configuration where you can select the note-type (news-items by default) and the content-types that should be displayed on the board. You can also enable adding new notes when clicking on the canvas and create an archive for outdated notes.

The advantages to websites that offer a similar features as a service are obvious:

  • The data lives only on your server, which is non-negotiable for intranets.
  • The code is open source, you can adapt it to your needs.

The board works with archetypes and dexterity-types. Yes, you can use it with!

Since the content has the default workflow, notes are usually not published after creating when using Plone out-of-the-box. We included an option to auto-publish new notes to work around this. Use it with care.

Most of the board-functionality is in javascript using backbone.js and jQueryUI. Feel free to copy that code and integrate it into other backends. To create sticky notes you'd only need a few lines of javascript and jQueryUI (see this jsfiddle for the full example):

$(function () {

Our javascript-code is a little more complex since we have to deal with a lot more UI, overlays, TinyMCE and so on.

The code can be found on GitHub. The package is on PyPi. developed collective.noticeboard for the German Institute for International Educational Research (DIPF).

Report from the Munich-leg of the Wine-and-Beer-Sprint

erstellt von Philip Bauer zuletzt geändert: 2013-01-30T13:39:12+01:00
tl;dr: Successful Plone Sprint roundup mentioning wormhole, beer, white sausages and huge progress in the plone.* namespace :)

Last week about 40 people met from Thursday to Sunday in Syslab's Office in Munich for the Wine and Beer Sprint to push Plone forward. I was awed by the large number of people willing to attend, we actually had to close the list a few days before the event since having more than 40 people in the office would have been too bad for air-quality. As it turned out, Syslabs office is a great location to host a sprint since it not only has one huge room but also several smaller ones where groups of people could work in conclave.

We had a mixed crowd of seasoned (2/3) and first-time (1/3) sprinters. What was especially great was the attendance of some people from the local Python user group (that recently merged with the Plone user group), one of them a django-guy who took his first steps with Plone.

Munich was frozen solid during the sprint, a fact that was driven home by the people from the other leg of the sprint (in Capetown, South Africa) who were sitting in the sun wearing t-shirts grinning at us through the wormhole we set up to link the sprints. From what we could understand (wormhole-technology is still only in alpha-state) they had a great time even though they could not get ice for their drinks by stepping outside.

The sprint was very productive, here are some noteable results:

  • The team working on managed to get a release-candidate out. The next-generation translation machinery (that works for Dexterity and Archetypes) is production-ready. At we've been using early stages of it in production together with for over a year now and are very happy.
  • Together with Timo Stollenwerk I was trying to get people to work on since I believe that Plone with dexterity-based default-types is a huge win for users (who can extend their types without paying a developer) and developers (who can focus on more challenging tasks than creating simple content-types). We managed to release 1.0b1, although not all features made it into the release and the documention needs more love. Thanks to the work of many (inluding Carsten Senger, Pavel Bogdanovic, Manuel Reinhardt, Tom Blockley, David Glick and my dear coworkers Patrick Gerken and Steffen Lindner) the package now has migrations from Archetypes, working indexes, robot-framework-tests and much more. Timo and me will work on a PLIP for it.
  • A group around the tireless Nejc Zupan managed to release Version 1.0rc1 of plone.api, an effort started at the Plone Konferenz 2012 in Munich (with a "k").
  • Some people worked on Patterns, a javascript-framework aiming to bring webdesign and development together. They fixed a ton of bugs and revamped their website. Rok Garbas worked on making patterns useable in Plone, most noteably and
  • Domen Kožar worked on mr.bob [], a successor to paster.
  • Besides working on Timo Stollenwerk also worked on getting rid of formlib and portal tools. Now all control panels have been migrated to either plone.autoform or Timo also held a tutorial on test-driven-development in which he started collective.ploneboard, a rewrite of Products.Ploneboard from scratch with Dexterity and Plus: Timo learned that Augustiner Edelstoff is really good beer :-)
  • and the Website of the Python Software Verband have been updated, enhanced and their code moved to github.
  • We saw a very promising sneak preview of new designs and mockups for and
  • After the last report-out with South-Africa I even managed to give a short plone-development tutorial by showing a workspace-ish behavior for Dexterity-folders, that sadly did not see any progress during the sprint.

The non-coding part of the sprint was also a success. The mountain of white sausages that we prepared for the Weisswurstfrühstück will stay in my memory forever. Contrary to the title of the sprint, coffee, Club Mate and Single Malt Whiskey were more popular than beer.

Thanks to Alex Pilz and Florian Friesdorf who not only volunteered their offices to the sprint but also did most of the organizing and were great hosts. Thanks to the Python Software Verband for sponsoring beverages and food.

The biggest thanks goes to the many sprinters who chose to spend their time to come to the sprint to improve Plone when they could have done so many other things. The spirit of friendship and the dedication of people in the Plone Community to give back continues to humble me and makes me glad and greatful to be part of that.

Last Preparations for PloneConf

erstellt von pgerken — zuletzt geändert: 2012-10-02T12:52:07+01:00
Philip and I are doing our last steps for the plone training. Also, we ask each other what else we could forget, did you miss any of these?

We are in the last steps for our two-day Training "Mastering Plone". As far as I can tell we'd still welcome more people. We just sent our a mail to the participants about how to prepare a virtual machine with vagrant for the conference. It installs Plone's unified installer in a ubuntu 12.4-box with puppet using the systems python 2.7. Hopefully everyone should have the same Plone running on their laptops even before the training starts on Monday morning.

Apart from that, don't forget:

  1. Internet in Hotels can be slow. It might be better to get a local card than to disable roaming on your phone. Here is a list of dutch sim card providers with data plans:
  2. Don't trust WiFi! From what I have heard, it has happened in the past that Wifi routers have been stolen during the conference, hacked for data sniffing and reinserted into the conference LAN, probably with key loggers. I am pretty sure you access your customers plone sites via https only (wink, wink!), but what about your phone apps, you know they won't transmit your password in clear text? Don't take any risks, use a VPN!
    I have had bad experiences with HideMyAss btw, as soon as I was in brazil, they thought my VPN account was used for fraud and asked for some details I didn't have with me, to not cancel my account. Apparently they want you to use HMA from home only.
  3. Sleep as much as you can before you go. You won't get any sleep during the conference.

See you in Arnhem!