All Versions
72
Latest Version
Avg Release Cycle
79 days
Latest Release
536 days ago

Changelog History
Page 6

  • v3.9.3 Changes

    October 23, 2009

    ๐Ÿ› Bugs Fixed

    • 2 BTree bugs, introduced by a bug fix in 3.9.0c2, sometimes caused deletion of keys to be improperly handled, resulting in data being available via iteraation but not item access.
  • v3.9.2 Changes

    October 13, 2009

    ๐Ÿ› Bugs Fixed

    • ZEO manages a separate thread for client network IO. It created this thread on import, which caused problems for applications that implemented daemon behavior by forking. Now, the client thread isn't created until needed.

    • File-storage pack clean-up tasks that can take a long time unnecessarily blocked other activity.

    • In certain rare situations, ZEO client connections would hang during the initial connection setup.

  • v3.9.1 Changes

    October 01, 2009

    ๐Ÿ› Bugs Fixed

    • Conflict errors committing blobs caused ZEO servers to stop committing transactions.
  • v3.9.0 Changes

    September 08, 2009

    ๐Ÿ†• New Features (in more or less reverse chronological order)

    • The Database class now has an xrefs keyword argument and a corresponding allow-implicit-cross-references configuration option. which default to true. When set to false, cross-database references are disallowed.

    • โž• Added support for RelStorage.

    • As a convenience, the connection root method for returning the root object can now also be used as an object with attributes mapped to the root-object keys.

    • Databases have a new method, transaction, that can be used with the Python (2.5 and later) with statement::

      db = ZODB.DB(...) with db.transaction() as conn: # ... do stuff with conn

    This uses a private transaction manager for the connection. If control exits the block without an error, the transaction is committed, otherwise, it is aborted.

    • Convenience functions ZODB.connection and ZEO.connection provide a convenient way to open a connection to a database. They open a database and return a connection to it. When the connection is closed, the database is closed as well.

    • ๐Ÿ‘ The ZODB.config databaseFrom... methods now support multi-databases. If multiple zodb sections are used to define multiple databases, the databases are connected in a multi-database arrangement and the first of the defined databases is returned.

    • The zeopack script has gotten a number of improvements:

      • Simplified command-line interface. (The old interface is still supported, except that support for ZEO version 1 servers has been dropped.)
      • Multiple storages can be packed in sequence.
      • This simplifies pack scheduling on servers serving multiple databases.
      • All storages are packed to the same time.
      • You can now specify a time of day to pack to.
      • The script will now time out if it can't connect to s storage in 60 seconds.
    • The connection now estimates the object size based on its pickle size and informs the cache about size changes.

    The database got additional configurations options (cache-size-bytes and historical-cache-size-bytes) to limit the cache size based on the estimated total size of cached objects. The default values are 0 which has the interpretation "do not limit based on the total estimated size". There are corresponding methods to read and set the new configuration parameters.

    • Connections now have a public opened attribute that is true when the connection is open, and false otherwise. When true, it is the seconds since the epoch (time.time()) when the connection was opened. This is a renaming of the previous _opened private variable.

    • ๐Ÿ‘ FileStorage now supports blobs directly.

    • You can now control whether FileStorages keep .old files when packing.

    • POSKeyErrors are no longer logged by ZEO servers, because they are really client errors.

    • ๐Ÿ‘ A new storage interface, IExternalGC, to support external garbage collection, http://wiki.zope.org/ZODB/ExternalGC, has been defined and implemented for FileStorage and ClientStorage.

    • โœ… As a small convenience (mainly for tests), you can now specify initial data as a string argument to the Blob constructor.

    • ZEO Servers now provide an option, invalidation-age, that allows quick verification of ZEO clients have been disconnected for less than a given time even if the number of transactions the client hasn't seen exceeds the invalidation queue size. This is only recommended if the storage being served supports efficient iteration from a point near the end of the transaction history.

    • ๐Ÿ‘ The FileStorage iterator now handles large files better. When iterating from a starting transaction near the end of the file, the iterator will scan backward from the end of the file to find the starting point. This enhancement makes it practical to take advantage of the new storage server invalidation-age option.

    • Previously, database connections were managed as a stack. This tended to cause the same connection(s) to be used over and over. For example, the most used connection would typically be the only connection used. In some rare situations, extra connections could be opened and end up on the top of the stack, causing extreme memory wastage. Now, when connections are placed on the stack, they sink below existing connections that have more active objects.

    • ๐Ÿ”ง There is a new pool-timeout database configuration option to specify that connections unused after the given time interval should be garbage collection. This will provide a means of dealing with extra connections that are created in rare circumstances and that would consume an unreasonable amount of memory.

    • ๐Ÿ‘ The Blob open method now supports a new mode, 'c', to open committed data for reading as an ordinary file, rather than as a blob file. The ordinary file may be used outside the current transaction and even after the blob's database connection has been closed.

    • ClientStorage now provides blob cache management. When using non-shared blob directories, you can set a target cache size and the cache will periodically be reduced try to keep it below the target size.

    The client blob directory layout has changed. If you have existing non-shared blob directories, you will have to remove them.

    • ZODB 3.9 ZEO clients can connect to ZODB 3.8 servers. ZODB ZEO clients from ZODB 3.2 on can connect to ZODB 3.9 servers.

    • When a ZEO cache is stale and would need verification, a ZEO.interfaces.StaleCache event is published (to zope.event). Applications may handle this event and take action such as exiting the application without verifying the cache or starting cold.

    • There's a new convenience function, ZEO.DB, for creating databases using ZEO Client Storages. Just call ZEO.DB with the same arguments you would otherwise pass to ZEO.ClientStorage.ClientStorage::

      import ZEO db = ZEO.DB(('some_host', 8200))

    • Object saves are a little faster

    • ๐Ÿ”ง When configuring storages in a storage server, the storage name now defaults to "1". In the overwhelmingly common case that a single storage, the name can now be omitted.

    • FileStorage now provides optional garbage collection. A 'gc' keyword option can be passed to the pack method. A false value prevents garbage collection.

    • The FileStorage constructor now provides a boolean pack_gc option, which defaults to True, to control whether garbage collection is performed when packing by default. This can be overridden with the gc option to the pack method.

    The ZConfig configuration for FileStorage now includes a pack-gc option, corresponding to the pack_gc constructor argument.

    • The FileStorage constructor now has a packer keyword argument that allows an alternative packer to be supplied.

    The ZConfig configuration for FileStorage now includes a packer option, corresponding to the packer constructor argument.

    • ๐Ÿ‘ MappingStorage now supports multi-version concurrency control and iteration and provides a better storage implementation example.

    • DemoStorage has a number of new features:

      • The ability to use a separate storage, such as a file storage to store changes
      • Blob support
      • Multi-version concurrency control and iteration
      • Explicit support for demo-storage stacking via push and pop methods.
    • Wen calling ZODB.DB to create a database, you can now pass a file name, rather than a storage to use a file storage.

    • โž• Added support for copying and recovery of blob storages:

      • Added a helper function, ZODB.blob.is_blob_record for testing whether a data record is for a blob. This can be used when iterating over a storage to detect blob records so that blob data can be copied.

      In the future, we may want to build this into a blob-aware iteration interface, so that records get blob file attributes automatically.

      • Added the IBlobStorageRestoreable interfaces for blob storages that support recovery via a restoreBlob method.
      • Updated ZODB.blob.BlobStorage to implement IBlobStorageRestoreable and to have a copyTransactionsFrom method that also copies blob data.
    • New ClientStorage configuration option drop_cache_rather_verify. If this option is true then the ZEO client cache is dropped instead of the long (unoptimized) verification. For large caches, setting this option can avoid effective down times in the order of hours when the connection to the ZEO server was interrupted for a longer time.

    • Cleaned-up the storage iteration API and provided an iterator implementation for ZEO.

    • ๐Ÿ”– Versions are no-longer supported.

    • ๐Ÿ‘€ Document conflict resolution (see ZODB/ConflictResolution.txt).

    • ๐Ÿ‘Œ Support multi-database references in conflict resolution.

    • ๐Ÿ‘‰ Make it possible to examine oid and (in some situations) database name of persistent object references during conflict resolution.

    • ๐Ÿšš Moved the 'transaction' module out of ZODB. ZODB depends upon this module, but it must be installed separately.

    • ZODB installation now requires setuptools.

    • โž• Added offset information to output of fstail script. Added test harness for this script.

    • โž• Added support for read-only, historical connections based on datetimes or serials (TIDs). See src/ZODB/historical_connections.txt.

    • โœ‚ Removed the ThreadedAsync module.

    • Now depend on zc.lockfile

    ๐Ÿ› Bugs Fixed

    • ๐Ÿ›  CVE-2009-2701: Fixed a vulnerability in ZEO storage servers when blobs are available. Someone with write access to a ZEO server configured to support blobs could read any file on the system readable by the server process and remove any file removable by the server process.

    • BTrees (and TreeSets) kept references to internal keys. https://bugs.launchpad.net/zope3/+bug/294788

    • ๐Ÿ‘ BTree Sets and TreeSets don't support the standard set add method. (Now either add or the original insert method can be used to add an object to a BTree-based set.)

    • ๐Ÿ”ง The runzeo script didn't work without a configuration file. (https://bugs.launchpad.net/zodb/+bug/410571)

    • ๐Ÿ—„ Officially deprecated PersistentDict (https://bugs.launchpad.net/zodb/+bug/400775)

    • Calling setstate on a persistent object could under certain uncommon cause the process to crash. (https://bugs.launchpad.net/zodb/+bug/262158)

    • When committing transactions involving blobs to ClientStorages with non-shared blob directories, a failure could occur in tpc_finish if there was insufficient disk space to copy the blob file or if the file wasn't available. https://bugs.launchpad.net/zodb/+bug/224169

    • ๐Ÿ’พ Savepoint blob data wasn't properly isolated. If multiple simultaneous savepoints in separate transactions modified the same blob, data from one savepoint would overwrite data for another.

    • ๐Ÿ’พ Savepoint blob data wasn't cleaned up after a transaction abort. https://bugs.launchpad.net/zodb/+bug/323067

    • Opening a blob with modes 'r+' or 'a' would fail when the blob had no committed changes.

    • PersistentList's sort method did not allow passing of keyword parameters. Changed its sort parameter list to match that of its (Python 2.4+) UserList base class.

    • Certain ZEO server errors could cause a client to get into a state where it couldn't commit transactions. https://bugs.launchpad.net/zodb/+bug/374737

    • ๐Ÿ›  Fixed vulnerabilities in the ZEO network protocol that allow:

      • CVE-2009-0668 Arbitrary Python code execution in ZODB ZEO storage servers
      • CVE-2009-0669 Authentication bypass in ZODB ZEO storage servers

    The vulnerabilities only apply if you are using ZEO to share a database among multiple applications or application instances and if untrusted clients are able to connect to your ZEO servers.

    • ๐Ÿ›  Fixed the setup test command. It previously depended on private functions in zope.testing.testrunner that don't exist any more.

    • ZEO client threads were unnamed, making it hard to debug thread management.

    • ๐Ÿ‘ ZEO protocol 2 support was broken. This caused very old clients to be unable to use new servers.

    • 0๏ธโƒฃ zeopack was less flexible than it was before. -h should default to local host.

    • 0๏ธโƒฃ The "lawn" layout was being selected by default if the root of the blob directory happened to contain a hidden file or directory such as ".svn". Now hidden files and directories are ignored when choosing the default layout.

    • BlobStorage was not compatible with MVCC storages because the wrappers were being removed by each database connection. Fixed.

    • Saving indexes for large file storages failed (with the error: RuntimeError: maximum recursion depth exceeded). This can cause a FileStorage to fail to start because it gets an error trying to save its index.

    • Sizes of new objects weren't added to the object cache size estimation, causing the object-cache size limiting feature to let the cache grow too large when many objects were added.

    • โœ‚ Deleted records weren't removed when packing file storages.

    • ๐Ÿ›  Fixed analyze.py and added test.

    • ๐Ÿ›  fixed Python 2.6 compatibility issue with ZEO/zeoserverlog.py

    • ๐Ÿ—„ using hashlib.sha1 if available in order to avoid DeprecationWarning under Python 2.6

    • made runzeo -h work

    • The monitor server didn't correctly report the actual number of clients.

    • Packing could return spurious errors due to errors notifying disconnected clients of new database size statistics.

    • ๐Ÿ”ง Undo sometimes failed for FileStorages configured to support blobs.

    • Starting ClientStorages sometimes failed with non-new but empty cache files.

    • The history method on ZEO clients failed.

    • ๐Ÿ›  Fix for bug #251037: Make packing of blob storages non-blocking.

    • ๐Ÿ›  Fix for bug #220856: Completed implementation of ZEO authentication.

    • ๐Ÿ›  Fix for bug #184057: Make initialisation of small ZEO client file cache sizes not fail.

    • ๐Ÿ›  Fix for bug #184054: MappingStorage used to raise a KeyError during load instead of a POSKeyError.

    • ๐Ÿ›  Fixed bug in Connection.TmpStore: load() would not defer to the backend storage for loading blobs.

    • ๐Ÿ›  Fix for bug #181712: Make ClientStorage update lastTransaction directly after connecting to a server, even when no cache verification is necessary.

    • ๐Ÿ›  Fixed bug in blob filesystem helper: the isSecure check was inverted.

    • ๐Ÿ›  Fixed bug in transaction buffer: a tuple was unpacked incorrectly in clear.

    • ๐Ÿ›  Bugfix the situation in which comparing persistent objects (for instance, as members in BTree set or keys of BTree) might cause data inconsistency during conflict resolution.

    • ๐Ÿ›  Fixed bug 153316: persistent and BTrees were using int for memory sizes which caused errors on x86_64 Intel Xeon machines (using 64-bit Linux).

    • ๐Ÿ›  Fixed small bug that the Connection.isReadOnly method didn't work after a savepoint.

    • ๐Ÿ› Bug #98275: Made ZEO cache more tolerant when invalidating current versions of objects.

    • ๐Ÿ›  Fixed a serious bug that could cause client I/O to stop (hang). This was accompanied by a critical log message along the lines of: "RuntimeError: dictionary changed size during iteration".

    • ๐Ÿ›  Fixed bug #127182: Blobs were subclassable which was not desired.

    • ๐Ÿ›  Fixed bug #126007: tpc_abort had untested code path that was broken.

    • ๐Ÿ›  Fixed bug #129921: getSize() function in BlobStorage could not deal with garbage files

    • ๐Ÿ›  Fixed bug in which MVCC would not work for blobs.

    • ๐Ÿ›  Fixed bug in ClientCache that occurred with objects larger than the total cache size.

    • ๐Ÿ”’ When an error occured attempting to lock a file and logging of said error was enabled.

    • FileStorages previously saved indexes after a certain number of writes. This was done during the last phase of two-phase commit, which made this critical phase more subject to errors than it should have been. Also, for large databases, saves were done so infrequently as to be useless. The feature was removed to reduce the chance for errors during the last phase of two-phase commit.

    • File storages previously kept an internal object id to transaction id mapping as an optimization. This mapping caused excessive memory usage and failures during the last phase of two-phase commit. This optimization has been removed.

    • โ™ป๏ธ Refactored handling of invalidations on ZEO clients to fix a possible ordering problem for invalidation messages.

    • On many systems, it was impossible to create more than 32K blobs. Added a new blob-directory layout to work around this limitation.

    • ๐Ÿ›  Fixed bug that could lead to memory errors due to the use of a Python dictionary for a mapping that can grow large.

    • ๐Ÿ›  Fixed bug #251037: Made packing of blob storages non-blocking.

    • ๐Ÿ›  Fixed a bug that could cause InvalidObjectReference errors for objects that were explicitly added to a database if the object was modified after a savepoint that added the object.

    • ๐Ÿ›  Fixed several bugs that caused ZEO cache corruption when connecting to servers. These bugs affected both persistent and non-persistent caches.

    • ๐Ÿ‘Œ Improved the the ZEO client shutdown support to try to avoid spurious errors on exit, especially for scripts, such as zeopack.

    • Packing failed for databases containing cross-database references.

    • Cross-database references to databases with empty names weren't constructed properly.

    • The zeo client cache used an excessive amount of memory, causing applications with large caches to exhaust available memory.

    • ๐Ÿ›  Fixed a number of bugs in the handling of persistent ZEO caches:

      • Cache records are written in several steps. If a process exits after writing begins and before it is finishes, the cache will be corrupt on restart. The way records are written was changed to make cache record updates atomic.
      • There was no lock file to prevent opening a cache multiple times at once, which would lead to corruption. Persistent caches now use lock files, in the same way that file storages do.
      • A bug in the cache-opening logic led to cache failure in the unlikely event that a cache has no free blocks.
    • When using ZEO Client Storages, Errors occured when trying to store objects too big to fit in the ZEO cache file.

    • ๐Ÿ›  Fixed bug in blob filesystem helper: the isSecure check was inverted.

    • ๐Ÿ›  Fixed bug in transaction buffer: a tuple was unpacked incorrectly in clear.

    • ๐Ÿ›  Fixed bug in Connection.TmpStore: load() would not defer to the back-end storage for loading blobs.

    • ๐Ÿ›  Fixed bug #190884: Wrong reference to POSKeyError caused NameError.

    • ๐Ÿ›  Completed implementation of ZEO authentication. This fixes issue 220856.

    What's new in ZODB 3.8.0

    General

    • ๐Ÿš€ (unreleased) Fixed setup.py use of setuptools vs distutils, so .c and .h files are included in the bdist_egg.

    • The ZODB Storage APIs have been documented and cleaned up.

    • ๐Ÿ—„ ZODB versions are now officially deprecated and support for them will be removed in ZODB 3.9. (They have been widely recognized as deprecated for quite a while.)

    • ๐Ÿ”„ Changed the automatic garbage collection when opening a connection to only apply the garbage collections on those connections in the pool that are closed. (This fixed issue 113923.)

    ZEO

    • (3.8a1) ZEO's strategoes for avoiding client cache verification were improved in the case that servers are restarted. Before, if transactions were committed after the restart, clients that were up to date or nearly up to date at the time of the restart and then connected had to verify their caches. Now, it is far more likely that a client that reconnects soon after a server restart won't have to verify its cache.

    • ๐Ÿ›  (3.8a1) Fixed a serious bug that could cause clients that disconnect from and reconnect to a server to get bad invalidation data if the server serves multiple storages with active writes.

    • (3.8a1) It is now theoretically possible to use a ClientStorage in a storage server. This might make it possible to offload read load from a storage server at the cost of increasing write latency. This should increase write throughput by offloading reads from the final storage server. This feature is somewhat experimental. It has tests, but hasn't been used in production.

    Transactions

    • (3.8a1) Add a doom() and isDoomed() interface to the transaction module.

    First step towards the resolution of http://www.zope.org/Collectors/Zope3-dev/655

    A doomed transaction behaves exactly the same way as an active transaction but raises an error on any attempt to commit it, thus forcing an abort.

    Doom is useful in places where abort is unsafe and an exception cannot be raised. This occurs when the programmer wants the code following the doom to run but not commit. It is unsafe to abort in these circumstances as a following get() may implicitly open a new transaction.

    Any attempt to commit a doomed transaction will raise a DoomedTransaction exception.

    • (3.8a1) Clean up the ZODB imports in transaction.

    Clean up weird import dance with ZODB. This is unnecessary since the transaction module stopped being imported in ZODB/init.py in rev 39622.

    • ๐Ÿšš (3.8a1) Support for subtransactions has been removed in favor of save points.

    Blobs

    • โšก๏ธ (3.8b1) Updated the Blob implementation in a number of ways. Some of these are backward incompatible with 3.8a1:

    o The Blob class now lives in ZODB.blob

    o The blob openDetached method has been replaced by the committed method.

    • ๐Ÿ‘€ (3.8a1) Added new blob feature. See the ZODB/Blobs directory for documentation.

    ZODB now handles (reasonably) large binary objects efficiently. Useful to use from a few kilobytes to at least multiple hundred megabytes.

    BTrees

    • ๐Ÿ‘ (3.8a1) Added support for 64-bit integer BTrees as separate types.

    (For now, we're retaining compile-time support for making the regular integer BTrees 64-bit.)

    • (3.8a1) Normalize names in modules so that BTrees, Buckets, Sets, and TreeSets can all be accessed with those names in the modules (e.g., BTrees.IOBTree.BTree). This is in addition to the older names (e.g., BTrees.IOBTree.IOBTree). This allows easier drop-in replacement, which can especially be simplify code for packages that want to support both 32-bit and 64-bit BTrees.

    • (3.8a1) Describe the interfaces for each module and actually declare the interfaces for each.

    • (3.8a1) Fix module references so klass.module points to the Python wrapper module, not the C extension.

    • (3.8a1) introduce module families, to group all 32-bit and all 64-bit modules.

  • v3.7.0 Changes

    ๐Ÿš€ Release date: 2007-04-20

    Packaging

    • ๐Ÿ“ฆ (3.7.0b3) ZODB is now packaged without it's dependencies

    ZODB no longer includes copies of dependencies such as ZConfig, zope.interface and so on. It now treats these as dependencies. If ZODB is installed with easy_install or zc.buildout, the dependencies will be installed automatically.

    • ๐Ÿ— (3.7.0b3) ZODB is now a buildout

    ZODB checkouts are now built and tested using zc.buildout.

    • ๐ŸŒฒ (3.7b4) Added logic to avoid spurious errors from the logging system on exit.

    • ๐Ÿ”€ (3.7b2) Removed the "sync" mode for ClientStorage.

    Previously, a ClientStorage could be in either "sync" mode or "async" mode. Now there is just "async" mode. There is now a dedicicated asyncore main loop dedicated to ZEO clients.

    Applications no-longer need to run an asyncore main loop to cause client storages to run in async mode. Even if an application runs an asyncore main loop, it is independent of the loop used by client storages.

    This addresses a test failure on Mac OS X, http://www.zope.org/Collectors/Zope3-dev/650, that I believe was due to a bug in sync mode. Some asyncore-based code was being called from multiple threads that didn't expect to be.

    Converting to always-async mode revealed some bugs that weren't caught before because the tests ran in sync mode. These problems could explain some problems we've seen at times with clients taking a long time to reconnect after a disconnect.

    Added a partial heart beat to try to detect lost connections that aren't otherwise caught, http://mail.zope.org/pipermail/zodb-dev/2005-June/008951.html, by perioidically writing to all connections during periods of inactivity.

    Connection management

    • (3.7a1) When more than pool_size connections have been closed, DB forgets the excess (over pool_size) connections closed first. Python's cyclic garbage collection can take "a long time" to reclaim them (and may in fact never reclaim them if application code keeps strong references to them), but such forgotten connections can never be opened again, so their caches are now cleared at the time DB forgets them. Most applications won't notice a difference, but applications that open many connections, and/or store many large objects in connection caches, and/or store limited resources (such as RDB connections) in connection caches may benefit.

    BTrees

    • ๐Ÿ‘Œ Support for 64-bit integer keys and values has been provided as a compile-time option for the "I" BTrees (e.g. IIBTree).

    ๐Ÿ“š Documentation

    • โœ… (3.7a1) Thanks to Stephan Richter for converting many of the doctest files to ReST format. These are now chapters in the Zope 3 apidoc too.

    IPersistent

    • (3.7a1) The documentation for _p_oid now specifies the concrete type of oids (in short, an oid is either None or a non-empty string).

    โœ… Testing

    • โœ… (3.7b2) Fixed test-runner output truncation.

    A bug was fixed in the test runner that caused result summaries to be omitted when running on Windows.

    Tools

    • ๐ŸŒฒ (3.7a1) The changeover from zLOG to the logging module means that some tools need to perform minimal logging configuration themselves. Changed the zeoup script to do so and thus enable it to emit error messages.

    BTrees

    Connection

    • (3.7a1) An optimization for loading non-current data (MVCC) was inadvertently disabled in _setstate(); this has been repaired.

    persistent

    After Commit hooks

    • (3.7a1) Transaction objects have a new method, addAfterCommitHook(hook, *args, **kws). Hook functions registered with a transaction are called after the transaction commits or aborts. For example, one might want to launch non transactional or asynchrnonous code after a successful, or aborted, commit. See test_afterCommitHook() in transaction/tests/test_transaction.py for a tutorial doctest, and the ITransaction interface for details.
  • v3.6.2 Changes

    ๐Ÿš€ Release date: 15-July-2006

    DemoStorage

    ๐Ÿš€ Following is combined news from internal releases (to support ongoing ๐Ÿš€ Zope2 / Zope3 development). These are the dates of the internal releases:

    • 3.6.1 27-Mar-2006
    • 3.6.0 05-Jan-2006
    • 3.6b6 01-Jan-2006
    • 3.6b5 18-Dec-2005
    • 3.6b4 04-Dec-2005
    • 3.6b3 06-Nov-2005
    • 3.6b2 25-Oct-2005
    • 3.6b1 24-Oct-2005
    • 3.6a4 07-Oct-2005
    • 3.6a3 07-Sep-2005
    • 3.6a2 06-Sep-2005
    • 3.6a1 04-Sep-2005

    ๐Ÿ—„ Removal of Features Deprecated in ZODB 3.4

    ๐Ÿ—„ (3.6b2) ZODB 3.6 no longer contains features officially deprecated in the ๐Ÿš€ ZODB 3.4 release. These include:

    • get_transaction(). Use transaction.get() instead. transaction.commit() is a shortcut spelling of transaction.get().commit(), and transaction.abort() of transaction.get().abort(). Note that importing ZODB no longer installs get_transaction as a name in Python's __builtin__ module either.

    • The begin() method of Transaction objects. Use the begin() method of a transaction manager instead. transaction.begin() is a shortcut spelling to call the default transaction manager's begin() method.

    • The dt argument to Connection.cacheMinimize().

    • The Connection.cacheFullSweep() method. Use cacheMinimize() instead.

    • The Connection.getTransaction() method. Pass a transaction manager to DB.open() instead.

    • The Connection.getLocalTransaction() method. Pass a transaction manager to DB.open() instead.

    • The cache_deactivate_after and version_cache_deactivate_after arguments to the DB constructor.

    • The temporary, force, and waitflag arguments to DB.open(). DB.open() no longer blocks (there's no longer a fixed limit on the number of open connections).

    • The transaction and txn_mgrarguments to DB.open(). Use the transaction_manager argument instead.

    • The getCacheDeactivateAfter, setCacheDeactivateAfter, getVersionCacheDeactivateAfter and setVersionCacheDeactivateAfter methods of DB.

    Persistent

    • โš  (3.6.1) Suppressed warnings about signedness of characters when compiling under GCC 4.0.x. See http://www.zope.org/Collectors/Zope/2027.

    • (3.6a4) ZODB 3.6 introduces a change to the basic behavior of Persistent objects in a particular end case. Before ZODB 3.6, setting obj._p_changed to a true value when obj was a ghost was ignored: obj remained a ghost, and getting obj._p_changed continued to return None. Starting with ZODB 3.6, obj is activated instead (unghostified), and its state is changed from the ghost state to the changed state. The new behavior is less surprising and more robust.

    • (3.6b5) The documentation for _p_oid now specifies the concrete type of oids (in short, an oid is either None or a non-empty string).

    Commit hooks

    • (3.6a1) The beforeCommitHook() method has been replaced by the new addBeforeCommitHook() method, with a more-robust signature. beforeCommitHook() is now deprecated, and will be removed in ZODB 3.8. Thanks to Julien Anguenot for contributing code and tests.

    Connection management

    • (3.6b6) When more than pool_size connections have been closed, DB forgets the excess (over pool_size) connections closed first. Python's cyclic garbage collection can take "a long time" to reclaim them (and may in fact never reclaim them if application code keeps strong references to them), but such forgotten connections can never be opened again, so their caches are now cleared at the time DB forgets them. Most applications won't notice a difference, but applications that open many connections, and/or store many large objects in connection caches, and/or store limited resources (such as RDB connections) in connection caches may benefit.

    ZEO

    • (3.6a4) Collector 1900. In some cases of pickle exceptions raised by low-level ZEO communication code, callers of marshal.encode() could attempt to catch an exception that didn't actually exist, leading to an erroneous AttributeError exception. Thanks to Tres Seaver for the diagnosis.

    BaseStorage

    • ๐Ÿ‘ป (3.6a4) Nothing done by tpc_abort() should raise an exception. However, if something does (an error case), BaseStorage.tpc_abort() left the commit lock in the acquired state, causing any later attempt to commit changes hang.

    Multidatabase

    • (3.6b1) The database_name for a database in a multidatabase collection can now be specified in a config file's <zodb> section, as the value of the optional new database_name key. The .databases attribute cannot be specified in a config file, but can be passed as the optional new databases argument to the open() method of a ZConfig factory for type ZODBDatabase. For backward compatibility, Zope 2.9 continues to allow using the name in its <zodb_db name> config section as the database name (note that <zodb_db> is defined by Zope, not by ZODB -- it's a Zope-specific extension of ZODB's <zodb> section).

    PersistentMapping

    • (3.6.1) PersistentMapping was inadvertently pickling volatile attributes (http://www.zope.org/Collectors/Zope/2052).

    • (3.6b4) PersistentMapping makes changes by a pop() method call persistent now (http://www.zope.org/Collectors/Zope/2036).

    • (3.6a1) The PersistentMapping class has an __iter__() method now, so that objects of this type work well with Python's iteration protocol. For example, if x is a PersistentMapping (or Python dictionary, or BTree, or PersistentDict, ...), then for key in x: iterates over the keys of x, list(x) creates a list containing x's keys, iter(x) creates an iterator for x's keys, and so on.

    Tools

    • ๐ŸŒฒ (3.6b5) The changeover from zLOG to the logging module means that some tools need to perform minimal logging configuration themselves. Changed the zeoup script to do so and thus enable it to emit error messages.

    BTrees

    • โš  (3.6.1) Suppressed warnings about signedness of characters when compiling under GCC 4.0.x. See http://www.zope.org/Collectors/Zope/2027.

    • 0๏ธโƒฃ (3.6a1) BTrees and Buckets now implement the setdefault() and pop() methods. These are exactly like Python's dictionary methods of the same names, except that setdefault() requires both arguments (and Python is likely to change to require both arguments too -- defaulting the default argument to None has no viable use cases). Thanks to Ruslan Spivak for contributing code, tests, and documentation.

    • (3.6a1) Collector 1873. It wasn't possible to construct a BTree or Bucket from, or apply their update() methods to, a PersistentMapping or PersistentDict. This works now.

    ZopeUndo

    • ๐Ÿ›  (3.6a4) Collector 1810. A previous bugfix (#1726) broke listing undoable transactions for users defined in a non-root acl_users folder. Zope logs a acl_users path together with a username (separated by a space) and this previous fix failed to take this into account.

    Connection

    • (3.6b5) An optimization for loading non-current data (MVCC) was inadvertently disabled in _setstate(); this has been repaired.

    ๐Ÿ“š Documentation

    • โœ… (3.6b3) Thanks to Stephan Richter for converting many of the doctest files to ReST format. These are now chapters in the Zope 3 apidoc too.

    • (3.6b4) Several misspellings of "occurred" were repaired.

    Development

    • (3.6a1) The source code for the old ExtensionClass-based Persistence package moved, from ZODB to the Zope 2.9 development tree. ZODB 3.5 makes no use of Persistence, and, indeed, the Persistence package could not be compiled from a ZODB release, since some of the C header files needed appear only in Zope.

    • (3.6a3) Re-added the zeoctl module, for the same reasons mkzeoinst was re-added (see below).

    • (3.6a2) The mkzeoinst module was re-added to ZEO, because Zope3 has a script that expects to import it from there. ZODB's mkzeoinst script was rewritten to invoke the mkzeoinst module.

    transact

    • (3.6b4) Collector 1959: The undocumented transact module no longer worked. It remains undocumented and untested, but thanks to Janko Hauser it's possible that it works again ;-).
  • v3.5.1 Changes

    ๐Ÿš€ Release date: 26-Sep-2005

    ๐Ÿš€ Following is combined news from internal releases (to support ongoing ๐Ÿš€ Zope3 development). These are the dates of the internal releases:

    • 3.5.1b2 07-Sep-2005
    • 3.5.1b1 06-Sep-2005

    ๐Ÿ— Build

    • (3.5.1b2) Re-added the zeoctl module, for the same reasons mkzeoinst was re-added (see below).

    • (3.5.1b1) The mkzeoinst module was re-added to ZEO, because Zope3 has a script that expects to import it from there. ZODB's mkzeoinst script was rewritten to invoke the mkzeoinst module.

    ZopeUndo

    • ๐Ÿ›  (3.5.1) Collector 1810. A previous bugfix (#1726) broke listing undoable transactions for users defined in a non-root acl_users folder. Zope logs a acl_users path together with a username (separated by a space) and this previous fix failed to take this into account.
  • v3.5.0 Changes

    ๐Ÿš€ Release date: 31-Aug-2005

    ๐Ÿš€ Following is combined news from internal releases (to support ongoing ๐Ÿš€ Zope3 development). These are the dates of the internal releases:

    • 3.5a7 11-Aug-2005
    • 3.5a6 04-Aug-2005
    • 3.5a5 19-Jul-2005
    • 3.5a4 14-Jul-2005
    • 3.5a3 17-Jun-2005
    • 3.5a2 16-Jun-2005
    • 3.5a1 10-Jun-2005

    ๐Ÿ’พ Savepoints

    • ๐Ÿ—„ (3.5.0) As for deprecated subtransaction commits, the intent was that making a savepoint would invoke incremental garbage collection on Connection memory caches, to try to reduce the number of objects in cache to the configured cache size. Due to an oversight, this didn't happen, and stopped happening for subtransaction commits too. Making a savepoint (or doing a subtransaction commit) does invoke cache gc now.

    • (3.5a3) When a savepoint is made, the states of objects modified so far are saved to a temporary storage (an instance of class TmpStore, although that's an internal implementation detail). That storage needs to implement the full storage API too, but was missing the loadBefore() method needed for MVCC to retrieve non-current revisions of objects. This could cause spurious errors if a transaction with a pending savepoint needed to fetch an older revision of some object.

    • ๐Ÿ“„ (3.5a4) The ISavepoint interface docs said you could roll back to a given savepoint any number of times (until the transaction ends, or until you roll back to an earlier savepoint's state), but the implementation marked a savepoint as invalid after its first use. The implementation has been repaired, to match the docs.

    ZEO client cache

    • (3.5a6) Two memory leaks in the ZEO client cache were repaired, a major one involving ZEO.cache.Entry objects, and a minor one involving empty lists.

    ๐Ÿ—„ Subtransactions are deprecated

    • ๐Ÿšš (3.5a4) Subtransactions are deprecated, and will be removed in ZODB 3.7. Use savepoints instead. Savepoints are more powerful, and code using subtransactions does not mix well with code using savepoints (a subtransaction commit forces all current savepoints to become unusable, so code using subtransactions can hurt newer code trying to use savepoints). In general, a subtransaction commit done just to free memory can be changed from::

      transaction.commit(1)

    to::

      transaction.savepoint(True)
    

    That is, make a savepoint, and forget it. As shown, it's best to pass True for the optional optimistic argument in this case: because there's no possibility of asking for a rollback later, there's no need to insist that all data managers support rollback.

    In rarer cases, a subtransaction commit is followed later by a subtransaction abort. In that case, change the initial::

      transaction.commit(1)
    

    to::

      sp = transaction.savepoint()
    

    and in place of the subtransaction abort::

      transaction.abort(1)
    

    roll back the savepoint instead::

      sp.rollback()
    
    • (3.5a4) Internal uses of subtransactions (transaction commit() or abort() passing a true argument) were rewritten to use savepoints instead.

    Multi-database

    • ๐Ÿ‘ (3.5a1) Preliminary support for persistent cross-database references has been added. See ZODB/cross-database-references.txt for an introduction.

    Tools

    • (3.5a6, 3.5a7) Collector #1847. The ZEO client cache tracing and simulation tools weren't updated to work with ZODB 3.3, and the introduction of MVCC required major reworking of the tracing and simulation code. These tools are in a working state again, although so far lightly tested on just a few applications. In doc/ZEO/, see the heavily revised trace.txt and cache.txt.

    • (3.5a5) Collector #1846: If an uncommitted transaction was found, fsrecover.py fell into an infinite loop.

    ๐Ÿ Windows

    • (3.5a6) As developed in a long thread starting at http://mail.zope.org/pipermail/zope/2005-July/160433.html there appears to be a race bug in the Microsoft Windows socket implementation, rarely visible in ZEO when multiple processes try to create an "asyncore trigger" simultaneously. Windows-specific code in ZEO/zrpc/trigger.py changed to work around this bug when it occurs.

    ThreadedAsync.LoopCallback

    • (3.5a5) This once again physically replaces Python's asyncore.loop function with its own loop function, because it turns out Zope relied on the seemingly unused LoopCallback.exit_status global, which was removed in the change described below. Python's asyncore.loop is again not invoked, so any breakpoints or debugging prints added to that are again "lost".

    • (3.5a4) This replaces Python's asyncore.loop function with its own, in order to get notified when loop() is first called. The signature of asyncore.loop changed in Python 2.4, but LoopCallback.loop's signature didn't change to match. The code here was repaired to be compatible with both old and new signatures, and also repaired to invoke Python's asyncore.loop() instead of replacing it entirely (so, for example, debugging prints added to Python's asyncore.loop won't be lost anymore).

    FileStorage

    • (3.5a4) Collector #1830. In some error cases when reading a FileStorage index, the code referenced an undefined global.

    • (3.5a4) Collector #1822. The undoLog() and undoInfo() methods were changed in 3.4a9 to return the documented results. Alas, some pieces of (non-ZODB) code relied on the actual behavior. When the first and last arguments are both >= 0, these methods now treat them as if they were Python slice indices, including the first index but excluding the last index. This matches former behavior, although it contradicts older ZODB UML documentation. The documentation in ZODB.interfaces.IStorageUndoable was changed to match the new intent.

    • (3.5a2) The _readnext() method now returns the transaction size as the value of the "size" key. Thanks to Dieter Maurer for the patch, from http://mail.zope.org/pipermail/zodb-dev/2003-October/006157.html. "This is very valuable when you want to spot strange transaction sizes via Zope's 'Undo' tab".

    BTrees

    • (3.5.a5) Collector 1843. When a non-integer was passed to a method like keys() of a Bucket or Set with integer keys, an internal error code was overlooked, leading to everything from "delayed errors" to segfaults. Such cases raise TypeError now, as intended.

    • (3.5a4) Collector 1831. The BTree minKey() and maxKey() methods gave a misleading message if no key satisfying the constraints existed in a non-empty tree.

    • (3.5a4) Collector 1829. Clarified that the minKey() and maxKey() methods raise an exception if no key exists satsifying the constraints.

    • ๐Ÿšš (3.5a4) The ancient convert.py script was removed. It was intended to convert "old" BTrees to "new" BTrees, but the "old" BTree implementation was removed from ZODB years ago.

  • v3.4.1 Changes

    ๐Ÿš€ Release date: 09-Aug-2005

    ๐Ÿš€ Following are dates of internal releases (to support ongoing Zope 2 ๐Ÿš€ development) since ZODB 3.4's last public release:

    • 3.4.1b5 08-Aug-2005
    • 3.4.1b4 07-Aug-2005
    • 3.4.1b3 04-Aug-2005
    • 3.4.1b2 02-Aug-2005
    • 3.4.1b1 26-Jul-2005
    • 3.4.1a6 19-Jul-2005
    • 3.4.1a5 12-Jul-2005
    • 3.4.1a4 08-Jul-2005
    • 3.4.1a3 02-Jul-2005
    • 3.4.1a2 29-Jun-2005
    • 3.4.1a1 27-Jun-2005

    ๐Ÿ’พ Savepoints

    • (3.4.1a1) When a savepoint is made, the states of objects modified so far are saved to a temporary storage (an instance of class TmpStore, although that's an internal implementation detail). That storage needs to implement the full storage API too, but was missing the loadBefore() method needed for MVCC to retrieve non-current revisions of objects. This could cause spurious errors if a transaction with a pending savepoint needed to fetch an older revision of some object.

    • ๐Ÿ“„ (3.4.1a5) The ISavepoint interface docs said you could roll back to a given savepoint any number of times (until the transaction ends, or until you roll back to an earlier savepoint's state), but the implementation marked a savepoint as invalid after its first use. The implementation has been repaired, to match the docs.

    • (3.4.1b4) Collector 1860: use an optimistic savepoint in ExportImport (there's no possiblity of rollback here, so no need to insist that the data manager support rollbacks).

    ZEO client cache

    • (3.4.1b3) Two memory leaks in the ZEO client cache were repaired, a major one involving ZEO.cache.Entry objects, and a minor one involving empty lists.

    Subtransactions

    • (3.4.1a5) Internal uses of subtransactions (transaction commit() or abort() passing a true argument) were rewritten to use savepoints instead. Application code is strongly encouraged to do this too: subtransactions are weaker, will be deprecated soon, and do not mix well with savepoints (when you do a subtransaction commit, all current savepoints are made unusable). In general, a subtransaction commit done just to free memory can be changed from::

      transaction.commit(1)

    to::

      transaction.savepoint(True)
    

    That is, make a savepoint, and forget it. As shown, it's best to pass True for the optional optimistic argument in this case: because there's no possibility of asking for a rollback later, there's no need to insist that all data managers support rollback.

    In rarer cases, a subtransaction commit is followed later by a subtransaction abort. In that case, change the initial::

      transaction.commit(1)
    

    to::

      sp = transaction.savepoint()
    

    and in place of the subtransaction abort::

      transaction.abort(1)
    

    roll back the savepoint instead::

      sp.rollback()
    

    FileStorage

    • (3.4.1a3) Collector #1830. In some error cases when reading a FileStorage index, the code referenced an undefined global.

    • (3.4.1a2) Collector #1822. The undoLog() and undoInfo() methods were changed in 3.4a9 to return the documented results. Alas, some pieces of (non-ZODB) code relied on the actual behavior. When the first and last arguments are both >= 0, these methods now treat them as if they were Python slice indices, including the first index but excluding the last index. This matches former behavior, although it contradicts older ZODB UML documentation. The documentation in ZODB.interfaces.IStorageUndoable was changed to match the new intent.

    • (3.4.1a1) The UndoSearch._readnext() method now returns the transaction size as the value of the "size" key. Thanks to Dieter Maurer for the patch, from http://mail.zope.org/pipermail/zodb-dev/2003-October/006157.html. "This is very valuable when you want to spot strange transaction sizes via Zope's 'Undo' tab".

    ThreadedAsync.LoopCallback

    • (3.4.1a6) This once again physically replaces Python's asyncore.loop function with its own loop function, because it turns out Zope relied on the seemingly unused LoopCallback.exit_status global, which was removed in the change described below. Python's asyncore.loop is again not invoked, so any breakpoints or debugging prints added to that are again "lost".

    • (3.4.1a1) This replaces Python's asyncore.loop function with its own, in order to get notified when loop() is first called. The signature of asyncore.loop changed in Python 2.4, but LoopCallback.loop's signature didn't change to match. The code here was repaired to be compatible with both old and new signatures, and also repaired to invoke Python's asyncore.loop() instead of replacing it entirely (so, for example, debugging prints added to Python's asyncore.loop won't be lost anymore).

    ๐Ÿ Windows

    • (3.4.1b2) As developed in a long thread starting at http://mail.zope.org/pipermail/zope/2005-July/160433.html there appears to be a race bug in the Microsoft Windows socket implementation, rarely visible in ZEO when multiple processes try to create an "asyncore trigger" simultaneously. Windows-specific code in ZEO/zrpc/trigger.py changed to work around this bug when it occurs.

    Tools

    • (3.4.1b1 thru 3.4.1b5) Collector #1847. The ZEO client cache tracing and simulation tools weren't updated to work with ZODB 3.3, and the introduction of MVCC required major reworking of the tracing and simulation code. These tools are in a working state again, although so far lightly tested on just a few applications. In doc/ZEO/, see the heavily revised trace.txt and cache.txt.

    • (3.4.1a6) Collector #1846: If an uncommitted transaction was found, fsrecover.py fell into an infinite loop.

    DemoStorage

    • (3.4.1a1) The implementation of undoLog() was wrong in several ways; repaired.

    BTrees

    • (3.4.1a6) Collector 1843. When a non-integer was passed to a method like keys() of a Bucket or Set with integer keys, an internal error code was overlooked, leading to everything from "delayed errors" to segfaults. Such cases raise TypeError now, as intended.

    • (3.4.1a4) Collector 1831. The BTree minKey() and maxKey() methods gave a misleading message if no key satisfying the constraints existed in a non-empty tree.

    • (3.4.1a3) Collector 1829. Clarified that the minKey() and maxKey() methods raise an exception if no key exists satsifying the constraints.

  • v3.4 Changes

    ๐Ÿš€ Release date: 09-Jun-2005

    ๐Ÿš€ Following is combined news from the "internal releases" (to support ongoing Zope 2.8 and Zope3 development) since the last public ZODB 3.4 ๐Ÿš€ release. These are the dates of the internal releases:

    • 3.4c2 06-Jun-2005
    • 3.4c1 03-Jun-2005
    • 3.4b3 27-May-2005
    • 3.4b2 26-May-2005

    Connection, DB

    • (3.4b3) .transaction_manager is now a public attribute of IDataManager, and is the instance of ITransactionManager used by the data manager as its transaction manager. There was previously no way to ask a data manager which transaction manager it was using. It's intended that transaction_manager be treated as read-only.

    • (3.4b3) For sanity, the txn_mgr argument to DB.open(), Connection.__init__(), and Connection._setDB() has been renamed to transaction_manager. txn_mgr is still accepted, but is deprecated and will be removed in ZODB 3.6. Any code that was using the private ._txn_mgr attribute of Connection will break immediately.

    Development

    • โœ… (3.4b2) ZODB's test.py is now a small driver for the shared zope.testing.testrunner. See the latter's documentation for command-line arguments.

    Error reporting

    • (3.4c1) In the unlikely event that referencesf() reports an unpickling error (for example, a corrupt database can cause this), the message it produces no longer contains unprintable characters.

    โœ… Tests

    • (3.4c2) checkCrossDBInvalidations suffered spurious failures too often on slow and/or busy machines. The test is willing to wait longer for success now.