DBM (computing)

In computing, a DBM is a library and file format providing fast, single-keyed access to data. A key-value database from the original Unix, dbm is an early example of a NoSQL system.^[1]^[2]^[3]

History

The original dbm library and file format was a simple database engine, originally written by Ken Thompson and released by AT&T in 1979. The name is a three-letter acronym for DataBase Manager, and can also refer to the family of database engines with APIs and features derived from the original dbm.

The dbm library stores arbitrary data by use of a single key (a primary key) in fixed-size buckets and uses hashing techniques to enable fast retrieval of the data by key.

The hashing scheme used is a form of extendible hashing, so that the hashing scheme expands as new buckets are added to the database, meaning that, when nearly empty, the database starts with one bucket, which is then split when it becomes full. The two resulting child buckets will themselves split when they become full, so the database grows as keys are added.

The dbm library and its derivatives are pre-relational databases – they manage associative arrays, implemented as on-disk hash tables. In practice, they can offer a more practical solution for high-speed storage accessed by key, as they do not require the overhead of connecting and preparing queries. This is balanced by the fact that they can generally only be opened for writing by a single process at a time. An agent daemon can handle requests from multiple processes, but introduces IPC overhead.

Implementations

The original AT&T dbm library has been replaced by its many successor implementations. Notable examples include:^[3]

ndbm ("new dbm"), based on the original dbm with some new features.
GDBM ("GNU dbm"), GNU rewrite of the library implementing ndbm features and its own interface. Also provides new features like crash tolerance for guaranteeing data consistency.^[4]^[5]
sdbm ("small dbm"), a public domain rewrite of dbm. It is a part of the standard distribution for Perl and is available as an external library for Ruby.^[6]^[7]
qdbm ("Quick Database Manager"), a higher-performance dbm employing many of the same techniques as Tokyo/Kyoto Cabinet. Written by the same author before they moved on to the cabinets.^[8]
tdb ("Trivial Database"), a simple database used by Samba that supports multiple writers. Has a gdbm-based API.^[9]
Berkeley DB, 1991 replacement of ndbm by Sleepycat Software (now Oracle) created to get around the AT&T Unix copyright on BSD. It features many extensions like parallelism, transactional control, hashing, and B-tree storage.
LMDB: copy-on-write memory-mapped B+ tree implementation in C with a Berkeley-style API.

The following databases are dbm-inspired, but they do not directly provide a dbm interface, even though it would be trivial to wrap one:

cdb ("constant database"), database by Daniel J. Bernstein, database files can only be created and read, but never be modified
Tkrzw, an Apache 2.0 licensed successor to Kyoto Cabinet and Tokyo Cabinet
WiredTiger: database with traditional row-oriented and column-oriented storage.

Availability

As of 2001, the ndbm implementation of DBM was standard on Solaris and IRIX, whereas gdbm is ubiquitous on Linux. The Berkeley DB implementations were standard on some free operating systems.^[2]^[10] After a change of licensing of the Berkeley DB to GNU AGPL in 2013, projects like Debian have moved to LMDB.^[11]

Reliability

A 2018 AFL fuzzing test against many DBM-family databases exposed many problems in implementations when it comes to corrupt or invalid database files. Only freecdb by Daniel J. Bernstein showed no crashes. The authors of gdbm, tdb, and lmdb were prompt to respond. Berkeley DB fell behind due to the sheer amount of other issues;^[10] the fixes would be irrelevant to open-source software users due to the licensing change locking them back on an old version.^[11]

References

^ Kew 2007, p. 80: "DBMs have been with us since the early days of computing, when the need for fast keyed lookups was recognized. The original DBM is a UNIX-based library and file format for fast, highly-scalable keyed access to data. It was followed (in order) by NDBM ('new DBM'), GDBM ('GNU DBM'), and the Berkeley DB. This last is by far the most advanced, and the only DBM under active development today. Nevertheless, all of the DBMs from NDBM onward provide the same core functionality used by most programs, including Apache. A minimal-implementation SDBM is also bundled with APR, and is available to applications along with the other DBMs.
Although NDBM is now old - like the city named New Town ('Neapolis') by the Greeks in about 600BC and still called Naples today - it remains the baseline DBM. NDBM was used by early Apache modules such as the Apache 1.x versions of mod_auth_dbm and mod_rewrite. Both GDBM and Berkeley DB provide NDBM emulations, and Linux distributions ship with one or other of these emulations in place of the 'real' NDBM, which is excluded for licensing reasons. Unfortunately, the various file formats are totally incompatible, and there are subtle differences in behaviour concerning database locking. These issues led a steady stream of Linux users to report problems with DBMs in Apache 1.x."
^ ^a ^b Hazel 2001, p. 500: "The most common [single-key] format is called DBM. Most modern versions of Unix have a DBM library installed as standard, though this is not true of some older systems. The two most common DBM libraries are ndbm (standard on Solaris and IRIX) and Berkeley DB Version 2 or 3 (standard on several free operating systems). Exim supports both of these, as well as the older Berkeley DB Version 1, gdbm, and tdb."
^ ^a ^b
Ladd & O'Donnell 2001, pp. 823–824: "Most UNIX systems have some kind of DBM database. DBM is a set of library routines that manages data files consisting of key and value pairs. The DBM routines control how users enter and retrieve information from the database. Although it isn't the most powerful mechanism for storing information, using DBM is a faster method of retrieving information than using a flat file. Because most UNIX sites use one of the DBM libraries, the tools you need to store your information in a DBM database are readily available.
Almost as many flavors of the DBM libraries exist as there are UNIX systems. Although most of these libraries are compatible with each other, they all basically work the same way...
A list follows of some of the most popular DBM libraries available:
- DBM - DBM stores the database in two files. The first has the extension .Pag and contains the bitmap. The second, which has the extension .Dir, contains the data.
- NDBM - NDBM is much like DBM but with a few additional features; it was written to provide better storage and retrieval methods. Also, NDBM enables you to open many databases, unlike DBM, in which you are allowed to have only one database open within your script. Like DBM, NDBM stores its information in two files using the extensions .Pag and .Dir.
- SDBM - SDBM comes with the Perl archive, which has been ported to many platforms. Therefore, you can use DBM databases as long as a version of Perl exists for your computer. SDBM was written to match the functions provided with NDBM, so portability of code shouldn't be a problem. Perl is available on just about all popular platforms.
- GDBM - GDBM is the GNU version of the DBM family of database routines. GDBM also enables you to cache data, reducing the time that it takes to write to the database. The database has no size limit; its size depends completely on your system's resources. GDBM database files have the extension .Db. Unlike DBM and NDBM, both of which use two files, GDBM only uses one file.
- Berkeley db - The Berkeley db expands on the original DBM routines significantly. The Berkeley db uses hashed tables the same as the other DBM databases, but the library also can create databases based on a sorted balanced binary tree (BTREE) and store information with a record line number (RECNO). The method that you use depends completely on how you want to store and retrieve the information from a database. Berkeley db creates only one file, which has no extension."
^ "Crash Tolerance". GDBM manual. Retrieved 3 October 2021.
^ "Crashproofing the Original NoSQL Key-Value Store". Retrieved 3 October 2021.
^ yigit, ozan. "sdbm.bun". cse.yorku.ca. Retrieved 8 May 2019.
^ "Ruby SDBM library". SDBM on Github. Note that Ruby used to ship SDBM in the standard distribution up until version 2.7, after which it was made available only as an external library, similarly to the DBM and GDBM libraries, removed from the standard library in Ruby 3.1.
^ "QDBM: Quick Database Manager". fallabs.com. 2006. Archived from the original on 2020-02-27. Retrieved 2020-02-27.
^ "tdb: Main Page". tdb.samba.org.
^ ^a ^b Debroux, Lionel (16 Jun 2018). "oss-security - Fun with DBM-type databases..." openwall.com.
^ ^a ^b Surý, Ondřej (19 June 2014). "New project goal: Get rid of Berkeley DB (post jessie)". debian-devel (Mailing list). Debian.

Bibliography

Hazel, Philip (2001). Exim: The Mail Transfer Agent. O'Reilly.
Ladd, Eric; O'Donnell, Jim (2001). Using XHTML, XML and Java 2: Platinum Edition. Que. ISBN 9780789724731.
Kew, Nick (2007). The Apache Modules Book: Application Development with Apache. Prentice Hall Professional. ISBN 9780132704502.

SDBM library @Apache
Matthew, Neil; Stones, Richard (2008). "Databases". Beginning Linux Programming. Wiley.
Olson, Michael A.; Bostic, Keith; Seltzer, Margo (1999). "Berkeley DB" (PDF). Proceedings of the FREENIX Track:1999 USENIX Annual Technical Conference.

[2007-kew-apache-1] Kew 2007, p. 80: "DBMs have been with us since the early days of computing, when the need for fast keyed lookups was recognized. The original DBM is a UNIX-based library and file format for fast, highly-scalable keyed access to data. It was followed (in order) by NDBM ('new DBM'), GDBM ('GNU DBM'), and the Berkeley DB. This last is by far the most advanced, and the only DBM under active development today. Nevertheless, all of the DBMs from NDBM onward provide the same core functionality used by most programs, including Apache. A minimal-implementation SDBM is also bundled with APR, and is available to applications along with the other DBMs.
Although NDBM is now old - like the city named New Town ('Neapolis') by the Greeks in about 600BC and still called Naples today - it remains the baseline DBM. NDBM was used by early Apache modules such as the Apache 1.x versions of mod_auth_dbm and mod_rewrite. Both GDBM and Berkeley DB provide NDBM emulations, and Linux distributions ship with one or other of these emulations in place of the 'real' NDBM, which is excluded for licensing reasons. Unfortunately, the various file formats are totally incompatible, and there are subtle differences in behaviour concerning database locking. These issues led a steady stream of Linux users to report problems with DBMs in Apache 1.x."

[2001-hazel-exim-2] Hazel 2001, p. 500: "The most common [single-key] format is called DBM. Most modern versions of Unix have a DBM library installed as standard, though this is not true of some older systems. The two most common DBM libraries are ndbm (standard on Solaris and IRIX) and Berkeley DB Version 2 or 3 (standard on several free operating systems). Exim supports both of these, as well as the older Berkeley DB Version 1, gdbm, and tdb."

[2001-ladd-odonell-xhtml-3] Ladd & O'Donnell 2001, pp. 823–824: "Most UNIX systems have some kind of DBM database. DBM is a set of library routines that manages data files consisting of key and value pairs. The DBM routines control how users enter and retrieve information from the database. Although it isn't the most powerful mechanism for storing information, using DBM is a faster method of retrieving information than using a flat file. Because most UNIX sites use one of the DBM libraries, the tools you need to store your information in a DBM database are readily available.
Almost as many flavors of the DBM libraries exist as there are UNIX systems. Although most of these libraries are compatible with each other, they all basically work the same way...
A list follows of some of the most popular DBM libraries available:
DBM - DBM stores the database in two files. The first has the extension .Pag and contains the bitmap. The second, which has the extension .Dir, contains the data.
NDBM - NDBM is much like DBM but with a few additional features; it was written to provide better storage and retrieval methods. Also, NDBM enables you to open many databases, unlike DBM, in which you are allowed to have only one database open within your script. Like DBM, NDBM stores its information in two files using the extensions .Pag and .Dir.
SDBM - SDBM comes with the Perl archive, which has been ported to many platforms. Therefore, you can use DBM databases as long as a version of Perl exists for your computer. SDBM was written to match the functions provided with NDBM, so portability of code shouldn't be a problem. Perl is available on just about all popular platforms.
GDBM - GDBM is the GNU version of the DBM family of database routines. GDBM also enables you to cache data, reducing the time that it takes to write to the database. The database has no size limit; its size depends completely on your system's resources. GDBM database files have the extension .Db. Unlike DBM and NDBM, both of which use two files, GDBM only uses one file.
Berkeley db - The Berkeley db expands on the original DBM routines significantly. The Berkeley db uses hashed tables the same as the other DBM databases, but the library also can create databases based on a sorted balanced binary tree (BTREE) and store information with a record line number (RECNO). The method that you use depends completely on how you want to store and retrieve the information from a database. Berkeley db creates only one file, which has no extension."

[4] DBM - DBM stores the database in two files. The first has the extension .Pag and contains the bitmap. The second, which has the extension .Dir, contains the data.

[5] NDBM - NDBM is much like DBM but with a few additional features; it was written to provide better storage and retrieval methods. Also, NDBM enables you to open many databases, unlike DBM, in which you are allowed to have only one database open within your script. Like DBM, NDBM stores its information in two files using the extensions .Pag and .Dir.

[6] SDBM - SDBM comes with the Perl archive, which has been ported to many platforms. Therefore, you can use DBM databases as long as a version of Perl exists for your computer. SDBM was written to match the functions provided with NDBM, so portability of code shouldn't be a problem. Perl is available on just about all popular platforms.

[7] GDBM - GDBM is the GNU version of the DBM family of database routines. GDBM also enables you to cache data, reducing the time that it takes to write to the database. The database has no size limit; its size depends completely on your system's resources. GDBM database files have the extension .Db. Unlike DBM and NDBM, both of which use two files, GDBM only uses one file.

[8] Berkeley db - The Berkeley db expands on the original DBM routines significantly. The Berkeley db uses hashed tables the same as the other DBM databases, but the library also can create databases based on a sorted balanced binary tree (BTREE) and store information with a record line number (RECNO). The method that you use depends completely on how you want to store and retrieve the information from a database. Berkeley db creates only one file, which has no extension."

[4] "Crash Tolerance". GDBM manual. Retrieved 3 October 2021.

[5] "Crashproofing the Original NoSQL Key-Value Store". Retrieved 3 October 2021.

[6] yigit, ozan. "sdbm.bun". cse.yorku.ca. Retrieved 8 May 2019.

[7] "Ruby SDBM library". SDBM on Github. Note that Ruby used to ship SDBM in the standard distribution up until version 2.7, after which it was made available only as an external library, similarly to the DBM and GDBM libraries, removed from the standard library in Ruby 3.1.

[8] "QDBM: Quick Database Manager". fallabs.com. 2006. Archived from the original on 2020-02-27. Retrieved 2020-02-27.

[9] "tdb: Main Page". tdb.samba.org.

[fuzz-10] Debroux, Lionel (16 Jun 2018). "oss-security - Fun with DBM-type databases..." openwall.com.

[deb-11] Surý, Ondřej (19 June 2014). "New project goal: Get rid of Berkeley DB (post jessie)". debian-devel (Mailing list). Debian.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]