Plone Connection Podcast: Episode 01 - Philip Bauer

erstellt von Philip Bauer zuletzt geändert: 2020-11-19T18:56:39+02:00
I was honoured to have been invited as the guest on the inaugural episode of the Plone Connection Podcast!

The Plone Connection Podcast is a monthly podcast produced by Six Feet Up. Every month, Six Feet Up's Director of Engineering T. Kim Nguyen sits down with a different member of the Plone Community and asks them about their work with or on the Plone CMS.

Many thank to my good friend Kim for doing this!

Your first Plone 6 Project

erstellt von Philip Bauer zuletzt geändert: 2020-11-18T15:25:39+02:00
In this talk I picked some tasks from the Mastering Plone Training and showed how you can meet the same requirements using Volto, the new React-based front end for Plone.

I've had the opportunity to give this talk at the most excellent Python Web Conference.

ZODB Database debugging

erstellt von Philip Bauer zuletzt geändert: 2020-08-25T09:25:32+02:00
Debugging issues in the ZODB is one of the hardest tasks and can drive developers to insanity. Especially after the migration of projects to newer Versions of Plone or Python these skills are needed. Here is most of what I learned about this topic.

The problem

The ZODB contains python objects serializes as pickles. When a object is loaded/used a pickle is deserialized ("unpickled") into a python object.

A ZODB can contain objects that cannot be loaded. Reasons for that may be:

  • Code could not be loaded that is required to unpickle the object (e.g. removed packages, modules or classes)
  • Objects are referenced but missing from the database (e.g. a blob is missing)
  • The objects contains invalid entries (e.g. a reference to a oid that uses a no longer supported format)

The most frequent issues are caused by:

  • Improperly uninstalled or removed packages (e.g. Archetypes, ATContentTypes, CMFDefault, PloneFormGen etc.)
  • Code has changed but not all objects that rely on that code as updated
  • Code was refactored and old imports are no longer working

You should not blame the migration to Python 3 for these issues! Many issues may already exists in their database before the migration but people usually do not run checks to find issues. After a migration to Python 3 most people check their database for the first time. This may be because the documentation on python3-migration recommends running the tool zodbverify.

Real problems may be revealed at that point, e.g when:

  • Packing the Database fails
  • Features fail

You can check you ZODB for problems using the package zodbverify. To solve each of the issues you need to be able to answer three questions:

  1. Which object is broken and what is the error?
  2. Where is the object and what uses it?
  3. How do I fix it?

In short these approaches to fixing exist:

  1. Ignore the errors
  2. Add zodbupgrade mappings
  3. Patch your python-path to work around the errors
  4. Replace broken objects with dummies
  5. Remove broken objects the hard way
  6. Find our what and where broken objects are and then fix or remove them safely

I will mostly focus on the last approach.

But before you spend a lot of time to investigate individual errors it would be a good idea to deal with the most frequent problems, especially IntIds and Relations (see the chapter "Frequent Culprits") below. In my experience these usually solved most issues.

Find out what is broken

Check your entire database

Use zodbverify to verify a ZODB by iterating and loading all records. zodbverify is available as a standalone script and as addon for plone.recipe.zope2instance. Use the newest version!

In the simplest form run it like this:

$ bin/zodbverify -f var/filestorage/Data.fs

It will return:

  • a list of types of errors
  • the number of occurences
  • all oids that raise that error on loading


zodbverify is only available for Plone 5.2 and later. For older Plone-Versions use the scripts and from the ZODB package:

$ ./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs
$ ./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs

The output of zodbverify might look like this abbreviated example from a medium-sized intranet (1GB Data.fs, 5GB blobstorage) that started with Plone 4 on Archetypes and was migrated to Plone 5.2 on Python 3 and Dexterity:

$ ./bin/zodbverify -f var/filestorage/Data.fs


INFO:zodbverify:Done! Scanned 163955 records.
Found 1886 records that could not be loaded.
Exceptions, how often they happened and which oids are affected:

ModuleNotFoundError: No module named 'Products.Archetypes': 1487
0x0e00eb 0x0e00ee 0x0e00ef 0x0e00f0 0x0e00f1 0x2b194b 0x2b194e 0x2b194f 0x2b1950 [...]

ModuleNotFoundError: No module named 'Products.PloneFormGen': 289
0x2b1940 0x2b1941 0x2b1942 0x2b1943 0x2b1944 0x2b1974 0x2b1975 0x2b1976 0x2b1977 [...]

AttributeError: module 'App.interfaces' has no attribute 'IPersistentExtra': 34
0x2c0a69 0x2c0a6b 0x2c0ab7 0x2c0ab9 0x2c555d [...] 0x35907f

ModuleNotFoundError: No module named 'Products.CMFDefault': 20
0x011e 0x011f 0x0120 0x0121 0x0122 0x0123 0x0124 0x0125 0x0126 0x0127 0x0128 0x0129 0x012a 0x012b 0x012c 0x012d 0x012e 0x012f 0x0130 0x0131

ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 20
0x3b1cde 0x3b1ce0 0x3b1ce4 0x3b1ce6 0x3b1ce9 0x3b1ceb 0x3b1cee 0x3b1cf0 0x3b1cf4 0x3b1cf6 0x3b1cf9 0x3b1cfb 0x3b1cfe 0x3b1d00 0x3b1d04 0x3b1d06 0x3b1d09 0x3b1d0b 0x3b1d0e 0x3b1d10

ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16
0x2c0a68 0x2c0a6a 0x2c555c 0x2c555e 0x2c560b 0x2c560d 0x2c5663 0x2c5665 0x2c571b 0x2c571d 0x2c5774 0x2c5776 0x2c5833 0x2c5835 0x33272d 0x33272f

ModuleNotFoundError: No module named 'fourdigits': 8
0x28030f 0x280310 0x280311 0x280312 0x280313 0x280314 0x280315 0x280316

ModuleNotFoundError: No module named 'Products.ATContentTypes': 4
0x0e00e9 0x0e011a 0x0e01b3 0x0e0cb3

AttributeError: module '' has no attribute 'IEventSettings': 3
0x2a712b 0x2a712c 0x2a712d

ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

ModuleNotFoundError: No module named '': 1

ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

You can see all different types of errors that appear and which objects are causing them. Objects are referenced by their oid in the ZODB. See the Appendix on how to deal with oids.

You can see that among other issues there are still a lot of references to Archetypes and PloneFormGen (I omitted the complete lists) even though both are no longer used in the site.

Before the summary the log dumps a huge list of errors that contain the pickle and the error:

Could not process unknown record 0x376b77 (b'\x00\x00\x00\x00\x007kw'):
INFO:zodbverify:b'\x80\x03cProducts.PloneFormGen.content.thanksPage\nFormThanksPage\nq\x00.\x80\x03}q\x01(X\x0c\x00\x00\x00showinsearchq\x02\x88X\n\x00\x00\x00_signatureq\x03C\x10\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7q\x04X\r\x00\x00\x00creation_dateq\x05cDateTime.DateTime\nDateTime\nq\x06)\x81q\x07GA\xd6\xdf_\xba\xd56"\x89X\x05\x00\x00\x00GMT+2q\x08\x87q\tbX\r\x00\x00\x00marshall_hookq\nNX\n\x00\x00\x00showFieldsq\x0b]q\x0cX\x02\x00\x00\x00idq\rX\t\x00\x00\x00thank-youq\x0eX\x11\x00\x00\x00_at_creation_flagq\x0f\x88X\x11\x00\x00\x00modification_dateq\x10h\x06)\x81q\x11GA\xd6\xdf_\xba\xd7\x15r\x89h\x08\x87q\x12bX\x05\x00\x00\x00titleq\x13X\x05\x00\x00\x00Dankeq\x14X\x0f\x00\x00\x00demarshall_hookq\x15NX\x0e\x00\x00\x00includeEmptiesq\x16\x88X\x0e\x00\x00\x00thanksEpilogueq\x17C\x08\x00\x00\x00\x00\x007k\xaaq\x18cProducts.Archetypes.BaseUnit\nBaseUnit\nq\x19\x86q\x1aQX\x07\x00\x00\x00showAllq\x1b\x88X\x12\x00\x00\x00_EtagSupport__etagq\x1cX\r\x00\x00\x00ts34951147.36q\x1dX\x0b\x00\x00\x00portal_typeq\x1eX\x0e\x00\x00\x00FormThanksPageq\x1fX\x0b\x00\x00\x00searchwordsq C\x08\x00\x00\x00\x00\x007k\xabq!h\x19\x86q"QX\x07\x00\x00\x00_at_uidq#X \x00\x00\x00a2d15a36a521471daf2b7005ff9dbc62q$X\r\x00\x00\x00at_referencesq%C\x08\x00\x00\x00\x00\x007k\xacq&cOFS.Folder\nFolder\nq\'\x86q(QX\x0e\x00\x00\x00thanksPrologueq)C\x08\x00\x00\x00\x00\x007k\xadq*h\x19\x86q+QX\x0f\x00\x00\x00noSubmitMessageq,C\x08\x00\x00\x00\x00\x007k\xaeq-h\x19\x86q.QX\x03\x00\x00\x00_mdq/C\x08\x00\x00\x00\x00\x007k\xafq0cPersistence.mapping\nPersistentMapping\nq1\x86q2QX\x12\x00\x00\x00__ac_local_roles__q3}q4X\x16\x00\x00\x00xxx@xxx.deq5]q6X\x05\x00\x00\x00Ownerq7asu.'
INFO:zodbverify:Traceback (most recent call last):
  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/", line 62, in verify_record
    class_info = unpickler.load()
  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/", line 62, in find_class
    return super(Unpickler, self).find_class(modulename, name)
ModuleNotFoundError: No module named 'Products.PloneFormGen'

Inspecting a single object

In this case the object with the oid 0x376b77 seems to be a FormThanksPage from Products.PloneFormGen. But wait! You deleted all of these, so where in the site is it?

If the offending object is normal content the solution is mostly simple. You can call obj.getPhysicalPath() to find out where it is. But ore often than not editing and saving will fix the problem. In other cases you might need to copy the content to a new item and delete the broken object.

But usually it is not simply content but something else. Here are some examples:

  • A annotation on a object or the portal
  • A relationvalue in the relatopn-catalog
  • A item in the IntId catalog
  • A old revision of content in CMFEditions
  • A configuration-entry in portal_properties or in portal_registry

The hardest part is to find out what and where the broken object actually is before removing or fixing it.

The reason for that is that a entry in the ZODB does not know about it's parent. Acquisition finds parents with obj.aq_parent() but many items are not-Acquisition-aware. Only the parents that reference objects know about them.

A object x could be the attribute some_object on object y but you will not see that by inspecting x. Only y knows that x is y.some_object.

A way to work around this is used by the script on ZODB. It allows you to list all incoming and outgoing references to a certain object.

With this you will see that x is referenced by y. With this information you can then inspect the object y and hopefully see how x is set on y.

More often than not y is again not a object in the content-hierarchy but maybe a BTree of sorts, a pattern that is frequently used for effective storage of many items. Then you need to find out the parent of y to be able to fix x.

And so forth. It can a couple of steps until you end up in a item that can be identified, e.g. portal_properties or RelationCatalog and usually only exists once in a database.

To make the process of finding this path less tedious I extended zodbverify in with a feature that will show you all parents and their parents in a way that allows you to see where in the tree is it.

Before we look at the path of 0x376b77 we'll inspect the object.

Pass the oid and the debug-flag -D to zodbverify with ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D:

$ ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D

INFO:zodbverify:Inspecting 0x376b77:
<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>
Object as dict:
{'__Broken_newargs__': (), '__Broken_state__': {'showinsearch': True, '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7', 'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'), 'marshall_hook': None, 'showFields': [], 'id': 'thank-you', '_at_creation_flag': True, 'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'), 'title': 'Danke', 'demarshall_hook': None, 'includeEmpties': True, 'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>, 'showAll': True, '_EtagSupport__etag': 'ts34951147.36', 'portal_type': 'FormThanksPage', 'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>, '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62', 'at_references': <Folder at at_references>, 'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>, 'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>, '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>, '__ac_local_roles__': {'': ['Owner']}}}
The object is 'obj'
[2] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/
-> pickle, state = storage.load(oid)

Even before you use the provided pdb to inspect it you can see that it is of the class persistent broken, a way of the ZODB to give you access to objects even though their class can no longer be imported.

You can now inspect it:

(Pdb++) obj
<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>
(Pdb++) pp obj.__dict__
{'__Broken_newargs__': (),
 '__Broken_state__': {'_EtagSupport__etag': 'ts34951147.36',
                      '__ac_local_roles__': {'': ['Owner']},
                      '_at_creation_flag': True,
                      '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62',
                      '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>,
                      '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7',
                      'at_references': <Folder at at_references>,
                      'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'),
                      'demarshall_hook': None,
                      'id': 'thank-you',
                      'includeEmpties': True,
                      'marshall_hook': None,
                      'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'),
                      'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>,
                      'portal_type': 'FormThanksPage',
                      'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>,
                      'showAll': True,
                      'showFields': [],
                      'showinsearch': True,
                      'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>,
                      'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>,
                      'title': 'Danke'}}

If you now choose to continue (by pressing c) zodbverify it will try to disassemble the pickle. That is very useful for in-depth debugging but out of the scope of this documentation.

Inspect the path of references

Now you know it is broken but you still don't know where this ominous FormThanksPage actually is.

Continue to let zodbverify find the path to the object:

INFO:zodbverify:Building a reference-tree of ZODB...
INFO:zodbverify:Created a reference-dict for 163955 objects.

This oid is referenced by:

INFO:zodbverify:0x376ada BTrees.IOBTree.IOBucket at level 1
INFO:zodbverify:0x28018c BTrees.IOBTree.IOBTree at level 2
INFO:zodbverify:0x280184 five.intid.intid.IntIds at level 3
INFO:zodbverify:0x1e five.localsitemanager.registry.PersistentComponents at level 4
INFO:zodbverify:0x11 Products.CMFPlone.Portal.PloneSite at level 5
INFO:zodbverify:0x01 OFS.Application.Application at level 6
INFO:zodbverify: 8< --------------- >8 Stop at root objects

INFO:zodbverify:0x02f6 persistent.mapping.PersistentMapping at level 7
INFO:zodbverify: 8< --------------- >8 Stop at root objects

INFO:zodbverify:0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at level 8
INFO:zodbverify: 8< --------------- >8 Stop at root objects

INFO:zodbverify:0x02f5 at level 6
INFO:zodbverify:0x02fa zope.ramcache.ram.RAMCache at level 7
INFO:zodbverify:0x02fd at level 8
INFO:zodbverify:0x338f13 at level 9
INFO:zodbverify:0x0303 BTrees.OOBTree.OOBTree at level 10
INFO:zodbverify:0x346961 at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify:0x02fe at level 9
INFO:zodbverify:0x034d plone.keyring.keyring.Keyring at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify:0x376864 BTrees.IOBTree.IOBucket at level 3
INFO:zodbverify:0x31049f BTrees.IOBTree.IOBucket at level 4
INFO:zodbverify:0x325823 BTrees.IOBTree.IOBucket at level 5
INFO:zodbverify:0x3984c8 BTrees.IOBTree.IOBucket at level 6
INFO:zodbverify:0x2cce9a BTrees.IOBTree.IOBucket at level 7
INFO:zodbverify:0x2c6669 BTrees.IOBTree.IOBucket at level 8
INFO:zodbverify:0x2c62b4 BTrees.IOBTree.IOBucket at level 9
INFO:zodbverify:0x2c44c1 BTrees.IOBTree.IOBucket at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify:0x377536 BTrees.OIBTree.OIBucket at level 2
INFO:zodbverify:0x376b14 BTrees.OIBTree.OIBucket at level 3
INFO:zodbverify:0x376916 BTrees.OIBTree.OIBucket at level 4
INFO:zodbverify:0x376202 BTrees.OIBTree.OIBucket at level 5
INFO:zodbverify:0x373fa7 BTrees.OIBTree.OIBucket at level 6
INFO:zodbverify:0x37363a BTrees.OIBTree.OIBucket at level 7
INFO:zodbverify:0x372f26 BTrees.OIBTree.OIBucket at level 8
INFO:zodbverify:0x372cc8 BTrees.OIBTree.OIBucket at level 9
INFO:zodbverify:0x36eb86 BTrees.OIBTree.OIBucket at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

INFO:zodbverify:0x407185 BTrees.OIBTree.OIBTree at level 10
INFO:zodbverify: 8< --------------- >8 Stop after level 10!

You can see from the logged messages that the FormThanksPage is in a IOBucket which again is in a IOBTree which is in a object of the class five.intid.intid.IntIds which is part if the component-registry in the Plone site.

This means there is a reference to a broken object in the IntId tool. How to solve all these is covered below in the chapter "Frequent Culprits".

Decide how and if to fix it

In this case the solution is clear (remove refs to broken objects from the intid tool). But that is only one approach.

Often the solution is not presented like this (the solution to intid was not obvious to me until I spent considerable time to investigate).

The following six options to deal with these problems exists. Spoiler: Option 6 is the best approach in most cases but the other also have valid use-cases.

Option 1: Ignoring the errors

I do that a lot. Especially old databases that were migrated all the may from Plone 2 or 3 up to the current version have issues. If these issues never appear during operation and if clients have no budget or interest in fixing them you can leave them be. If they do not hurt you (e.g. you cannot pack your database or features actually fail) you can choose to ignore them.

At some point later they might appear and it may be a better time to fix them. I spent many hours fixing issues that will never show during operation.

Option 2: Migrating/Fixing a DB with zodbupdate

Use that when a module or class has moved or was renamed.


You can change objects in DB according to rules:

  • When a import has moved use a rename mapping
  • To specify if a obj needs to be decoded decode mapping

Examples from Zope/src/OFS/

zodbupdate_decode_dict = {
    'OFS.Image File data': 'binary',
    'OFS.Image Image data': 'binary',

    'OFS.Application Application title': 'utf-8',
    'OFS.DTMLDocument DTMLDocument title': 'utf-8',
    'OFS.DTMLMethod DTMLMethod title': 'utf-8',
    'OFS.DTMLMethod DTMLMethod raw': 'utf-8',
    'OFS.Folder Folder title': 'utf-8',
    'OFS.Image File title': 'utf-8',
    'OFS.Image Image title': 'utf-8',
    'OFS.Image Pdata title': 'utf-8',
    'OFS.Image Pdata data': 'binary',
    'OFS.OrderedFolder OrderedFolder title': 'utf-8',
    'OFS.userfolder UserFolder title': 'utf-8',

zodbupdate_rename_dict = {
    'webdav.LockItem LockItem': 'OFS.LockItem LockItem',

You can specify your own mappings in your own packages. These mappings need to be registered in so zodbupdate will pick them up.

Rename mapping example:

Decode mapping example:

Option 3: Work around with a patch

You can inject a module to work around missing or moved classes or modules.

The reason to want do this is usually because then you can safely delete items after that. They don't hurt your performance.

Examples in

# -*- coding: utf-8 -*-
from OFS.SimpleItem import SimpleItem
from import alias_module
from import bbb
from zope.interface import Interface

class IBBB(Interface):

class BBB(object):

SlideshowDescriptor = SimpleItem

# Interfaces
    from collective.z3cform.widgets.interfaces import ILayer
except ImportError:
    alias_module('collective.z3cform.widgets.interfaces.ILayer', IDummy)

    from App.interfaces import IPersistentExtra
except ImportError:
    alias_module('App.interfaces.IPersistentExtra', IDummy)

    from webdav.interfaces import IDAVResource
except ImportError:
    alias_module('webdav.interfaces.IDAVResource', IDummy)

# SimpleItem
    from collective.easyslideshow.descriptors import SlideshowDescriptor
except ImportError:
    alias_module('collective.easyslideshow.descriptors.SlideshowDescriptor', SlideshowDescriptor)

# object
    from collective.solr import interfaces
except ImportError:
    alias_module('collective.solr.indexer.SolrIndexProcessor', BBB)

    from Products.CMFPlone import UndoTool
except ImportError:
    sys.modules['Products.CMFPlone.UndoTool'] = bbb


Plone has plenty of these (see

Option 4: Replace broken objects with a dummy

If a objects is missing (i.e. you get a POSKeyError) or broken beyond repair you can choose to replace it with a dummy.

from persistent import Persistent
from ZODB.utils import p64
import transaction

app = self.context.__parent__
broken_oids = [0x2c0ab6, 0x2c0ab8]

for oid in broken_oids:
    dummy = Persistent()
    dummy._p_oid = p64(oid)
    dummy._p_jar = app._p_jar
    app._p_jar._added[dummy._p_oid] = dummy

You shoud be aware that the missing or broken object will be gone forever after you didi this. So before you choose to go down this path you should try to find out what the object in question actually was.

Option 5: Remove broken objects from db

from persistent import Persistent
from ZODB.utils import p64
import transaction

app = self.context.__parent__
broken_oids = [0x2c0ab6, 0x2c0ab8]

for oid in broken_oids:
    root = connection.root()
    del app._p_jar[p64(oid)]

I'm not sure if that is a acceptable approach under any circumstance since this will remove the pickle but not all references to the object. It will probably lead to PosKeyErrors.

Option 6: Manual fixing

This is how you should deal with most problems.

The way to go

  1. Use zodbverify to get all broken objects
  2. Pick one error-type at a time
  3. Use zodbverify with -o <OID> -D to inspect one object and find out where that object is referenced
  4. If you use follow referenced by until you find where in the tree the object lives. zodbverify will try to do it for you.
  5. Remove or fix the object (using a upgrade-step, pdb or a rename mapping)

Find out which items are broken

The newest version of zodbverify has a feature to that does the same task we discussed in Example 1 for you. Until it is merged and released you need to use the branch show_references from the pull-request

When inspecting a individual oid zodbverify builds a dict of all references for reverse-lookup. Then it recursively follow the trail of references to referencing items up to the root. To prevent irrelevant and recursive entries it aborts after level 600 and at some root-objects because these usually references a lot and would clutter the result with irrelevant information.

The output should give you a pretty good idea where in the object-tree a item is actually located, how to access and fix it.

If 0x3b1d06 is the broken oid inspect it with zodbverify:

$ ./bin/instance zodbverify -o 0x3b1d06 -D

2020-08-24 12:19:32,441 INFO    [Zope:45][MainThread] Ready to handle requests
2020-08-24 12:19:32,442 INFO    [zodbverify:222][MainThread]
The object is 'obj'
The Zope instance is 'app'
[4] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/
-> pickle, state = storage.load(oid)

(Pdb++) obj
<BTrees.OIBTree.OITreeSet object at 0x110b97ac0 oid 0x3b1d06 in <Connection at 10c524040>>

(Pdb++) pp [i for i in obj]
[<InterfaceClass OFS.EtagSupport.EtagBaseInterface>,
 <class 'webdav.interfaces.IDAVResource'>,
 <InterfaceClass plone.dexterity.interfaces.IDexterityContent>,
 <SchemaClass plone.supermodel.model.Schema>]

The problem now is that obj has no __parent__ so you have no way of knowing what you're actually dealing with.

When you press c for continue zodbverify will proceed and load the pickle:

(Pdb++) c
2020-08-24 12:20:50,784 INFO    [zodbverify:68][MainThread]
Could not process <class 'BTrees.OIBTree.OITreeSet'> record 0x3b1d06 (b'\x00\x00\x00\x00\x00;\x1d\x06'):
2020-08-24 12:20:50,784 INFO    [zodbverify:69][MainThread] b'\x80\x03cBTrees.OIBTree\nOITreeSet\nq\x00.\x80\x03(cOFS.EtagSupport\nEtagBaseInterface\nq\x01cAcquisition.interfaces\nIAcquirer\nq\\nIAllowDiscussion\nq\x03czope.annotation.interfaces\nIAnnotatable\nq\x04czope.annotation.interfaces\nIAttributeAnnotatable\nq\x05cplone.uuid.interfaces\nIAttributeUUID\nq\x06cProducts.CMFDynamicViewFTI.interfaces\nIBrowserDefault\nq\x07cProducts.CMFCore.interfaces\nICatalogAware\nq\x08cProducts.CMFCore.interfaces\nICatalogableDublinCore\nq\tczope.location.interfaces\nIContained\nq\ncProducts.CMFCore.interfaces\nIContentish\nq\x0bcOFS.interfaces\nICopySource\nq\x0ccwebdav.interfaces\nIDAVResource\nq\rcplone.dexterity.interfaces\nIDexterityContent\nq\\nIDexterityHasRelations\nq\x0fcplone.dexterity.interfaces\nIDexterityItem\nq\\nIDexterityIterateAware\nq\x11cplone.dexterity.interfaces\nIDexteritySchema\nq\\nIDocument\nq\x13cProducts.CMFCore.interfaces\nIDublinCore\nq\x14cProducts.CMFCore.interfaces\nIDynamicType\nq\\nIExcludeFromNavigation\nq\x16cz3c.relationfield.interfaces\nIHasIncomingRelations\nq\x17cz3c.relationfield.interfaces\nIHasOutgoingRelations\nq\x18cz3c.relationfield.interfaces\nIHasRelations\nq\x19cplone.namedfile.interfaces\nIImageScaleTraversable\nq\x1acOFS.interfaces\nIItem\nq\\nIIterateAware\nq\x1ccplone.portlets.interfaces\nILocalPortletAssignable\nq\x1dczope.location.interfaces\nILocation\nq\x1ecOFS.interfaces\nIManageable\nq\x1fcProducts.CMFCore.interfaces\nIMinimalDublinCore\nq cProducts.CMFCore.interfaces\nIMutableDublinCore\nq!cProducts.CMFCore.interfaces\nIMutableMinimalDublinCore\nq"\nINameFromTitle\nq#cApp.interfaces\nINavigation\nq$cProducts.CMFCore.interfaces\nIOpaqueItemManager\nq%cAccessControl.interfaces\nIOwned\nq&cAccessControl.interfaces\nIPermissionMappingSupport\nq\'cpersistent.interfaces\nIPersistent\nq(cOFS.interfaces\nIPropertyManager\nq)\nIRelatedItems\nq*cAccessControl.interfaces\nIRoleManager\nq+cplone.contentrules.engine.interfaces\nIRuleAssignable\nq,cProducts.CMFDynamicViewFTI.interfaces\nISelectableBrowserDefault\nq-cOFS.interfaces\nISimpleItem\\nITableOfContents\nq/cOFS.interfaces\nITraversable\nq0cplone.uuid.interfaces\nIUUIDAware\nq1cProducts.CMFEditions.interfaces\nIVersioned\\nIVersioningSupport\nq3cProducts.CMFCore.interfaces\nIWorkflowAware\nq4cOFS.interfaces\nIWriteLock\nq5cOFS.interfaces\nIZopeObject\nq6czope.interface\nInterface\nq7cplone.dexterity.schema.generated\nPlone_0_Document\nq8cplone.supermodel.model\nSchema\nq9tq:\x85q;\x85q<\x85q=.'
2020-08-24 12:20:50,786 INFO    [zodbverify:70][MainThread] Traceback (most recent call last):
  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/", line 64, in verify_record
  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/", line 62, in find_class
    return super(Unpickler, self).find_class(modulename, name)
ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package

    0: \x80 PROTO      3
    2: (    MARK
    3: c        GLOBAL     'OFS.EtagSupport EtagBaseInterface'
   38: q        BINPUT     1
   40: c        GLOBAL     'Acquisition.interfaces IAcquirer'
   74: q        BINPUT     2
   76: c        GLOBAL     ' IAllowDiscussion'
  135: q        BINPUT     3
  137: c        GLOBAL     'zope.annotation.interfaces IAnnotatable'
  178: q        BINPUT     4
  180: c        GLOBAL     'zope.annotation.interfaces IAttributeAnnotatable'
  230: q        BINPUT     5
  232: c        GLOBAL     'plone.uuid.interfaces IAttributeUUID'
  270: q        BINPUT     6
  272: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces IBrowserDefault'
  327: q        BINPUT     7
  329: c        GLOBAL     'Products.CMFCore.interfaces ICatalogAware'
  372: q        BINPUT     8
  374: c        GLOBAL     'Products.CMFCore.interfaces ICatalogableDublinCore'
  426: q        BINPUT     9
  428: c        GLOBAL     'zope.location.interfaces IContained'
  465: q        BINPUT     10
  467: c        GLOBAL     'Products.CMFCore.interfaces IContentish'
  508: q        BINPUT     11
  510: c        GLOBAL     'OFS.interfaces ICopySource'
  538: q        BINPUT     12
  540: c        GLOBAL     'webdav.interfaces IDAVResource'
  572: q        BINPUT     13
  574: c        GLOBAL     'plone.dexterity.interfaces IDexterityContent'
  620: q        BINPUT     14
  622: c        GLOBAL     ' IDexterityHasRelations'
  681: q        BINPUT     15
  683: c        GLOBAL     'plone.dexterity.interfaces IDexterityItem'
  726: q        BINPUT     16
  728: c        GLOBAL     ' IDexterityIterateAware'
  791: q        BINPUT     17
  793: c        GLOBAL     'plone.dexterity.interfaces IDexteritySchema'
  838: q        BINPUT     18
  840: c        GLOBAL     ' IDocument'
  885: q        BINPUT     19
  887: c        GLOBAL     'Products.CMFCore.interfaces IDublinCore'
  928: q        BINPUT     20
  930: c        GLOBAL     'Products.CMFCore.interfaces IDynamicType'
  972: q        BINPUT     21
  974: c        GLOBAL     ' IExcludeFromNavigation'
 1040: q        BINPUT     22
 1042: c        GLOBAL     'z3c.relationfield.interfaces IHasIncomingRelations'
 1094: q        BINPUT     23
 1096: c        GLOBAL     'z3c.relationfield.interfaces IHasOutgoingRelations'
 1148: q        BINPUT     24
 1150: c        GLOBAL     'z3c.relationfield.interfaces IHasRelations'
 1194: q        BINPUT     25
 1196: c        GLOBAL     'plone.namedfile.interfaces IImageScaleTraversable'
 1247: q        BINPUT     26
 1249: c        GLOBAL     'OFS.interfaces IItem'
 1271: q        BINPUT     27
 1273: c        GLOBAL     ' IIterateAware'
 1317: q        BINPUT     28
 1319: c        GLOBAL     'plone.portlets.interfaces ILocalPortletAssignable'
 1370: q        BINPUT     29
 1372: c        GLOBAL     'zope.location.interfaces ILocation'
 1408: q        BINPUT     30
 1410: c        GLOBAL     'OFS.interfaces IManageable'
 1438: q        BINPUT     31
 1440: c        GLOBAL     'Products.CMFCore.interfaces IMinimalDublinCore'
 1488: q        BINPUT     32
 1490: c        GLOBAL     'Products.CMFCore.interfaces IMutableDublinCore'
 1538: q        BINPUT     33
 1540: c        GLOBAL     'Products.CMFCore.interfaces IMutableMinimalDublinCore'
 1595: q        BINPUT     34
 1597: c        GLOBAL     ' INameFromTitle'
 1642: q        BINPUT     35
 1644: c        GLOBAL     'App.interfaces INavigation'
 1672: q        BINPUT     36
 1674: c        GLOBAL     'Products.CMFCore.interfaces IOpaqueItemManager'
 1722: q        BINPUT     37
 1724: c        GLOBAL     'AccessControl.interfaces IOwned'
 1757: q        BINPUT     38
 1759: c        GLOBAL     'AccessControl.interfaces IPermissionMappingSupport'
 1811: q        BINPUT     39
 1813: c        GLOBAL     'persistent.interfaces IPersistent'
 1848: q        BINPUT     40
 1850: c        GLOBAL     'OFS.interfaces IPropertyManager'
 1883: q        BINPUT     41
 1885: c        GLOBAL     ' IRelatedItems'
 1933: q        BINPUT     42
 1935: c        GLOBAL     'AccessControl.interfaces IRoleManager'
 1974: q        BINPUT     43
 1976: c        GLOBAL     'plone.contentrules.engine.interfaces IRuleAssignable'
 2030: q        BINPUT     44
 2032: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces ISelectableBrowserDefault'
 2097: q        BINPUT     45
 2099: c        GLOBAL     'OFS.interfaces ISimpleItem'
 2127: q        BINPUT     46
 2129: c        GLOBAL     ' ITableOfContents'
 2196: q        BINPUT     47
 2198: c        GLOBAL     'OFS.interfaces ITraversable'
 2227: q        BINPUT     48
 2229: c        GLOBAL     'plone.uuid.interfaces IUUIDAware'
 2263: q        BINPUT     49
 2265: c        GLOBAL     'Products.CMFEditions.interfaces IVersioned'
 2309: q        BINPUT     50
 2311: c        GLOBAL     ' IVersioningSupport'
 2370: q        BINPUT     51
 2372: c        GLOBAL     'Products.CMFCore.interfaces IWorkflowAware'
 2416: q        BINPUT     52
 2418: c        GLOBAL     'OFS.interfaces IWriteLock'
 2445: q        BINPUT     53
 2447: c        GLOBAL     'OFS.interfaces IZopeObject'
 2475: q        BINPUT     54
 2477: c        GLOBAL     'zope.interface Interface'
 2503: q        BINPUT     55
 2505: c        GLOBAL     'plone.dexterity.schema.generated Plone_0_Document'
 2556: q        BINPUT     56
 2558: c        GLOBAL     'plone.supermodel.model Schema'
 2589: q        BINPUT     57
 2591: t        TUPLE      (MARK at 2)
 2592: q    BINPUT     58
 2594: \x85 TUPLE1
 2595: q    BINPUT     59
 2597: \x85 TUPLE1
 2598: q    BINPUT     60
 2600: \x85 TUPLE1
 2601: q    BINPUT     61
 2603: .    STOP
highest protocol among opcodes = 2

If you are into this you can read the pickle now :)

If you press c again zodbverify will build the refernce-tree for this object and ispect if for you:

(Pdb++) c
2020-08-24 12:22:42,596 INFO    [zodbverify:234][MainThread] ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 0x3b1d06
2020-08-24 12:22:42,597 INFO    [zodbverify:43][MainThread] Building a reference-tree of ZODB...
2020-08-24 12:22:42,964 INFO    [zodbverify:60][MainThread] Objects: 10000
2020-08-24 12:22:44,167 INFO    [zodbverify:60][MainThread] Objects: 20000
2020-08-24 12:22:44,521 INFO    [zodbverify:60][MainThread] Objects: 30000
2020-08-24 12:22:44,891 INFO    [zodbverify:60][MainThread] Objects: 40000
2020-08-24 12:22:45,184 INFO    [zodbverify:60][MainThread] Objects: 50000
2020-08-24 12:22:45,507 INFO    [zodbverify:60][MainThread] Objects: 60000
2020-08-24 12:22:45,876 INFO    [zodbverify:60][MainThread] Objects: 70000
2020-08-24 12:22:46,403 INFO    [zodbverify:60][MainThread] Objects: 80000
2020-08-24 12:22:46,800 INFO    [zodbverify:60][MainThread] Objects: 90000
2020-08-24 12:22:47,107 INFO    [zodbverify:60][MainThread] Objects: 100000
2020-08-24 12:22:47,440 INFO    [zodbverify:60][MainThread] Objects: 110000
2020-08-24 12:22:47,747 INFO    [zodbverify:60][MainThread] Objects: 120000
2020-08-24 12:22:48,052 INFO    [zodbverify:60][MainThread] Objects: 130000
2020-08-24 12:22:48,375 INFO    [zodbverify:60][MainThread] Objects: 140000
2020-08-24 12:22:48,665 INFO    [zodbverify:60][MainThread] Objects: 150000
2020-08-24 12:22:48,923 INFO    [zodbverify:60][MainThread] Objects: 160000
2020-08-24 12:22:49,037 INFO    [zodbverify:61][MainThread] Created a reference-dict for 163955 objects.

2020-08-24 12:22:49,386 INFO    [zodbverify:182][MainThread] Save reference-cache as /Users/pbauer/.cache/zodbverify/zodb_references_0x03d7f331f3692266.json
2020-08-24 12:22:49,424 INFO    [zodbverify:40][MainThread] The oid 0x3b1d06 is referenced by:

0x3b1d06 (BTrees.OIBTree.OITreeSet) is referenced by 0x3b1d01 (BTrees.OOBTree.OOBucket) at level 1
0x3b1d01 (BTrees.OOBTree.OOBucket) is referenced by 0x11c284 (BTrees.OOBTree.OOBTree) at level 2
0x11c284 (BTrees.OOBTree.OOBTree) is _reltoken_name_TO_objtokenset for 0x11c278 (z3c.relationfield.index.RelationCatalog) at level 3
0x11c278 (z3c.relationfield.index.RelationCatalog) is relations for 0x1e (five.localsitemanager.registry.PersistentComponents) at level 4
0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x11 (Products.CMFPlone.Portal.PloneSite) at level 5
0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x01 (OFS.Application.Application) at level 6
8< --------------- >8 Stop at root objects
0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f6 (persistent.mapping.PersistentMapping) at level 7
8< --------------- >8 Stop at root objects
0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f7 (zope.component.persistentregistry.PersistentAdapterRegistry) at level 8
8< --------------- >8 Stop at root objects
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02f5 ( at level 6
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fa (zope.ramcache.ram.RAMCache) at level 7
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fd ( at level 8
0x02fd ( is __parent__ for 0x338f13 ( at level 9
0x338f13 ( is ++rule++rule-2 for 0x0303 (BTrees.OOBTree.OOBTree) at level 10
0x02fd ( is __parent__ for 0x346961 ( at level 10
0x02fd ( is __parent__ for 0x346b59 ( at level 11
0x02fd ( is __parent__ for 0x346b61 ( at level 12
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fe ( at level 9
0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x034d (plone.keyring.keyring.Keyring) at level 10
0x034d (plone.keyring.keyring.Keyring) is referenced by 0x02fb (persistent.mapping.PersistentMapping) at level 11
0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a32 (plone.keyring.keyring.Keyring) at level 12
0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a33 (plone.keyring.keyring.Keyring) at level 13
0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x3b3dc4 (pas.plugins.ldap.plonecontrolpanel.cache.CacheSettingsRecordProvider) at level 11
0x3b1d01 (BTrees.OOBTree.OOBucket) is _next for 0x3b1cf1 (BTrees.OOBTree.OOBucket) at level 3

From this output you can find out that the broken object is (surprise) a item in the RelationCatalog of zc.relation. See the chapter "Frequent Culprits" for information how to deal with these.

Example 1 of using

In this and the next example I will use the script to find out where a broken objects actually sits so I can remove or fix it. The easier approach is to use zodbverify but I discuss this approach here since it was your best option until I extended zodbverify and since it might help you to understand the way references work in the ZODB.

$ ./bin/zodbverify -f var/filestorage/Data.fs

INFO:zodbverify:Done! Scanned 120797 records.
Found 116 records that could not be loaded.
Exceptions and how often they happened:
AttributeError: Cannot find dynamic object factory for module plone.dexterity.schema.generated: 20
AttributeError: module '' has no attribute 'IEventSettings': 3
ModuleNotFoundError: No module named 'Products.ATContentTypes': 4
ModuleNotFoundError: No module named 'Products.Archetypes': 5
ModuleNotFoundError: No module named 'Products.CMFDefault': 20
ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1
ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1
ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1
ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1
ModuleNotFoundError: No module named 'fourdigits': 8
ModuleNotFoundError: No module named '': 2
ModuleNotFoundError: No module named ''; '' is not a package: 34
ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

Follow the white rabbit...

./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x35907d

oid 0x35907d BTrees.OIBTree.OISet 1 revision
    tid 0x03c425bfb4d8dcaa offset=282340 2017-12-15 10:07:42.386043
        tid user=b'Plone'
        tid description=b'/Plone/it-service/hilfestellungen-anleitungen-faq/outlook/content-checkout'
        new revision BTrees.OIBTree.OISet at 282469
    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413
        tid user=b''
        tid description=b''
        referenced by 0x35907b BTrees.OIBTree.OITreeSet at 911946038


Follow referenced by ...

./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x35907b

referenced by 0x3c5790 BTrees.OOBTree.OOBucket
./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x3c5790

referenced by 0x11c284 BTrees.OOBTree.OOBTree
./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x11c284

referenced by 0x3d0bd6 BTrees.OOBTree.OOBucket
./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x3d0bd6

referenced by 0x11c278 z3c.relationfield.index.RelationCatalog

Found it!!!!!

Example 2 of using

In this example zodbverify found a trace of Products.PloneFormGen even though you think you safely uninstalled the addon (e.g. using

Then find out where exists in the tree by following the trail of items that reference it:

./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x372d00
oid 0x372d00 Products.PloneFormGen.content.thanksPage.FormThanksPage 1 revision
    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413
        tid user=b''
        tid description=b''
        new revision Products.PloneFormGen.content.thanksPage.FormThanksPage at 912841984
        referenced by 0x372f26 BTrees.OIBTree.OIBucket at 912930339
        references 0x372e59 Products.Archetypes.BaseUnit.BaseUnit at 912841984
        references 0x372e5a Products.Archetypes.BaseUnit.BaseUnit at 912841984
        references 0x372e5b OFS.Folder.Folder at 912841984
        references 0x372e5c Products.Archetypes.BaseUnit.BaseUnit at 912841984
        references 0x372e5d Products.Archetypes.BaseUnit.BaseUnit at 912841984
        references 0x372e5e Persistence.mapping.PersistentMapping at 912841984
    tid 0x03d40a3e52a41633 offset=921078960 2019-11-25 17:02:19.368976
        tid user=b'Plone pbauer'
        tid description=b'/Plone/rename_file_ids'
        referenced by 0x2c1b51 BTrees.IOBTree.IOBucket at 921653012

Follow referenced by until you find something...

./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x2c1b51
oid 0x2c1b51 BTrees.IOBTree.IOBucket 1 revision

Here I skip the trail of referenced by until I find 0x280184 five.intid.intid.IntIds:

./bin/zopepy ./parts/packages/ZODB/scripts/ var/filestorage/Data.fs 0x280184
oid 0x280184 five.intid.intid.IntIds 1 revision
    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413
        tid user=b''
        tid description=b''
        new revision five.intid.intid.IntIds at 8579054
        references 0x28018c <unknown> at 8579054
        references 0x28018d <unknown> at 8579054
    tid 0x03d3e90c4d3aed55 offset=915868610 2019-11-19 19:24:18.100824
        tid user=b' adminstarzel'
        tid description=b'/Plone/portal_quickinstaller/installProducts'
        referenced by 0x02f6 persistent.mapping.PersistentMapping at 915868690
        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394
        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394
        referenced by 0x1e five.localsitemanager.registry.PersistentComponents at 915898834

That is the IntId-Catalog from zope.intid. The problem seems to be that similar to the zc.relation catalog rerefences to broken objects stay in the catalog and need to be removed manually.

Here is a example of how to remove all broken objects from the catalog in a pdb-session:

(Pdb++) from zope.intid.interfaces import IIntIds
(Pdb++) from zope.component import getUtility
(Pdb++) intid = getUtility(IIntIds)
(Pdb++) broken_keys = [i for i in intid.ids if 'broken' in repr(i.object)]
(Pdb++) for broken_key in broken_keys: intid.unregister(broken_key)
(Pdb++) import transaction
(Pdb++) transaction.commit()

After packing the DB the problem is gone. o/

Other Options

Use zodbbrowser to inspect the ZODB.
It is Zope3 app to navigate a ZODB in a browser. At least I had problems getting it to run with a Plone-ZODB.
Use zc.zodbdgc
This tool can validate distributed databases by starting at their root and traversing to make sure all referenced objects are reachable. Optionally, a database of reference information can be generated.
Use collective.zodbdebug
A great tool to build and inspect reference-maps and backreference-maps of a ZODB. So for it does not work with Python 3 yet. Some if its features are also part of zodbverify.

Frequent Culprits

IntIds and Relations

The IntId-Tool and the relation-catalog are by far the most requent issues, especially if you migrated from Archetypes to Dexterity.

There may be a lot of RelationValues in these Tools that still reference objects that cannot be loadedif these removed objects were not properly removed.

The following code from collective.relationhelpers cleans up the IntId- and Relation-catalog but keeps relations intact. For large sites it may take a while to run because it also needs to recreate linkintegrity-relations.

from collective.relationhelpers.api import cleanup_intids
from collective.relationhelpers.api import purge_relations
from collective.relationhelpers.api import restore_relations
from collective.relationhelpers.api import store_relations

def remove_relations(context=None):
    # store all relations in a annotation on the portal
    # empty the relation-catalog
    # remove all relationvalues and refs to broken objects from intid
    # recreate all relations from a annotation on the portal

For details see


Many addons and features in Plone store data in Annotations on the portal or on content.

It's a good idea to check IAnnotations(portal).keys() after a migration for Annotation that you can safely remove.

Here is a example where wicked (the now-removed wiki-style-editing feature of Plone) stored it's settings in a Annotation:

def cleanup_wicked_annotation(context=None):
    ann = IAnnotations(portal)
    if '' in ann:
        del ann['']

Another example is files from failed uploads stored by plone.formwidget.namedfile in a annotation:

def cleanup_upload_annotation(context=None):
    # remove traces of aborted uploads
    ann = IAnnotations(portal)
    if ann.get('file_upload_map', None) is not None:
        for uuid in ann['file_upload_map']:
            del ann['file_upload_map'][uuid]


Migrating a ZODB from py2 to py3

Since people often encounter issues with their ZODB after migrating here is a quick dive into migrating a ZODB from Python 2 to Python 3.

The migration is basically calling the script zodbupdate in py3 with the parameter --convert-py3.

$ ./bin/zodbupdate --convert-py3

You need to pass it the location of the database, the defaul-encoding (utf8) and a fallback-encoding (latin1) for items where decoding to utf8 fails.


$ ./bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding=utf8 --encoding-fallback latin1

Updating magic marker for var/filestorage/Data.fs
Ignoring index for /Users/pbauer/workspace/projectx/var/filestorage/Data.fs
Loaded 2 decode rules from AccessControl:decodes
Loaded 12 decode rules from OFS:decodes
Loaded 2 decode rules from Products.PythonScripts:decodes
Loaded 1 decode rules from Products.ZopeVersionControl:decodes
Committing changes (#1).

After that you should be able to use your ZODB in Python 3.

The process in a nutshell:

  1. First, run bin/zodbupdate -f var/filestorage/Data.fs So no python3 convert stuff yet! This will detect and apply several explicit and implicit rename rules.
  2. Then run bin/instance zodbverify. If this still gives warnings or exceptions, you may need to define more rules and apply them with zodbupdate. But you can still choose to migrate to py3 if this shows errors.
  3. Using Python 3 run bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding utf8
  4. For good measure, on Python 3 run bin/instance zodbverify.

Read the docs:

See also:

Dealing with oids

Transforming oids from int to hex and text and vice versa:

>>> from ZODB.utils import p64
>>> oid = 0x2c0ab6
>>> p64(oid)

>>> from ZODB.utils import oid_repr
>>> oid = b'\x00\x00\x00\x00\x00,\n\xb6'
>>> oid_repr(oid)

>>> from ZODB.utils import repr_to_oid
>>> oid = '0x2c0ab6'
>>> repr_to_oid(oid)

Get a path for blobs:

from ZODB.blob import BushyLayout
if isinstance(oid, int):
    # e.g. oid = 0x2c0ab6
    from ZODB.utils import p64
    oid = p64(oid)
return BushyLayout.oid_to_path(None, oid)

Load a obj by oid from ZODB in a pdb-prompt:

oid = 0x2c0ab6
from ZODB.utils import p64
app = self.context.__parent__
obj = app._p_jar.get(p64(oid))


Finish, merge and document


Quite a lot of people from the Plone/Zope communities have wriotten about this issue. I learned a lot from these posts:

I got interviewed about Plone 5.2

erstellt von Philip Bauer zuletzt geändert: 2019-07-19T16:15:53+02:00
I got asked a bunch of more or less relevant questions regarding Plone, Python 3 and the future.

Click here to read the interview on

Four options to try Plone 5.2 on Python 3

erstellt von Philip Bauer zuletzt geändert: 2019-03-08T09:40:10+02:00
Try these different options to get started with Plone on Python 3.

Demo pages

There are nightly build of the current Plone 5.2 coredev (the development version) with Python 2 and 3:

Minimal Buildout

Here is a minimal buildout to run Plone 5.2rc1 on Python 3:

parts = instance
extends =

recipe = plone.recipe.zope2instance
eggs =

You set it up like this:

$ python3.7 -m venv .
$ ./bin/pip install -r
$ ./bin/buildout

And start it as usual with ./bin/instance fg

Standalone Development buildout

extends =

parts =

eggs =

test-eggs =
    collective.easyform [test]

auto-checkout =

extensions =

show-picked-versions = true

recipe = plone.recipe.zope2instance
user = admin:admin
eggs = ${buildout:eggs}
debug-mode = on
verbose-security = on

recipe = zc.recipe.egg
eggs =
interpreter = zopepy
scripts =

recipe = collective.recipe.omelette
ignore-develop = False
eggs = ${buildout:eggs}
ignores = roman

recipe = collective.xmltestreport
eggs = ${buildout:test-eggs}
defaults = ['--auto-color', '--auto-progress']

recipe = zc.recipe.egg
eggs =

collective.easyform = git branch=python3


Starzel buildout

The buildout that we at use supports Plone 5.2rc1 with Python 2 and 3.

It has some nice features:

  • It extends to config- and version-files on github shared by all projects that use the same version of Plone.
  • It allows to update a project simply by changing the version it extends.
  • It allows to update all projects of one version by changing remote files (very useful for HotFixes).
  • It is minimal work to setup a new project.
  • It has presets for development, testing, staging and production.
  • It has all the nice development-helpers we use.


$ git clone -b 5.2rc1.x <SOME_PROJECT>

Remove all files that are not needed for a project but are only used for the buildout itself.

$ rm -rf linkto README.rst README.txt .travis.yml secret.cfg_tmpl VERSION.txt local_coredev.cfg CHANGES.rst

If you're not developing the buildout itself you want a create a new git repo.

$ rm -rf .git && git init

Add a file that contains a passwort. Do not use admin as a password in production!

$ echo -e "[buildout]\nlogin = admin\npassword = admin" > secret.cfg

Symlink to the file that best fits you local environment. At first that is usually development. Later you can use production or test. This buildout only uses local.cfg and ignores all local_*.cfg.

$ ln -s local_develop.cfg local.cfg

Create a virtualenv in Python 2.7 or Python 3.7 (Plone 5.2 only).

$ virtualenv .  # for Python 2.7
$ python3.7 -m venv .  # for Python 3 (Plone 5.2 only)

Install and configure Plone

$ ./bin/pip install -r requirements.txt
$ ./bin/buildout


I hope these options help you get started with Python 3. For serious projects you will likely create you own buildout.

Update (4.2.2019)

You should use the Plone coredev for developing Plone itself:

$ git clone coredev
$ cd coredev
$ git checkout 5.2
$ python3.7 -m venv .
$ ./bin/pip install -r requirements.txt
$ ./bin/buildout
$ ./bin/instance fg

Update 2 (24.2.2019)

Since Plone 5.2b2 wsgi is the default, even when running Python 2. See

This means you do not need to enable wsgi = on since it will be enabled automatically. You can still choose to switch wsgi = off in Python 2 if you have a good reason to do so (e.g. you need FTP or WebDAV). I updated the examples accordingly.

Update 3 (8.3.2019)

Release candidate 1 of Plone 5.2 is pending. See I updated the examples accordingly.