Repair Damaged RPM Database

If you encounter any of the following messages:

rpmdb: PANIC: fatal region error detected; run recovery
error: db3 error(22) from dbenv->open: Invalid argument
error: cannot open Packages index using db3 - Invalid argument (22)
error: cannot open Packages database in /var/lib/rpm
error: rpmdb: damaged header #1439 retrieved -- skipping.

or a similar RPM-related error message, it is probable that your RPM database is damaged.

RPM uses a database to maintain the package dependency information. This database is normally located at /var/lib/rpm.

# cd /var/lib/rpm
# ls -al
total 86420
drwxr-xr-x.  2 root root     4096 Nov 18 21:09 .
drwxr-xr-x. 45 root root     4096 Aug 25  2013 ..
-rw-r--r--.  1 root root  5820416 Sep  8  2013 Basenames
-rw-r--r--.  1 root root    16384 Aug 25  2013 Conflictname
-rw-r--r--.  1 root root   237568 Nov 18 21:30 __db.001
-rw-r--r--.  1 root root    73728 Nov 18 21:30 __db.002
-rw-r--r--.  1 root root  1318912 Nov 18 21:30 __db.003
-rw-r--r--.  1 root root  2695168 Sep  8  2013 Dirnames
-rw-r--r--.  1 root root    28672 Sep  8  2013 Group
-rw-r--r--.  1 root root    28672 Sep  8  2013 Installtid
-rw-r--r--.  1 root root    73728 Sep  8  2013 Name
-rw-r--r--.  1 root root    36864 Aug 19  2013 Obsoletename
-rw-r--r--.  1 root root 76402688 Sep  8  2013 Packages
-rw-r--r--.  1 root root   921600 Sep  8  2013 Providename
-rw-r--r--.  1 root root   647168 Sep  8  2013 Requirename
-rw-r--r--.  1 root root        0 Aug 18  2013 .rpm.lock
-rw-r--r--.  1 root root   172032 Sep  8  2013 Sha1header
-rw-r--r--.  1 root root   102400 Sep  8  2013 Sigmd5
-rw-r--r--.  1 root root     8192 Aug 19  2013 Triggername

RPM uses Berkeley DB, either version 3 or 4, for its database engine. Each of the Berkeley DB subsystems within an environment is described by one or more shared memory regions. Regions contain the per-process and per-thread shared information, including mutexes, that comprise a Berkeley DB environment.

Any files created in the filesystem to back the regions are created in the environment home directory specified to the DB open call. These files are named __db.###, e.g. __db.001, __db.002 and so on. When region files are backed by the filesystem, as in RPM, one file per region is created.

By the way, statistics about these regions can be displayed using the db_stat utility:

# cd /var/lib/rpm
# db_stat -e
Thu Dec 10 23:01:10 2015	Local time
0x120897	Magic number
0	Panic value
5.2.36	Environment version
9	Btree version
9	Hash version
1	Lock version
18	Log version
4	Queue version
2	Sequence version
1	Txn version
Wed Nov 18 21:09:09 2015	Creation time
0xcfca7f84	Environment ID
2	Primary region allocation and reference count mutex [0/10269 0% !Own]
1	References
232KB	Current region size
808KB	Maximum region size
Thread tracking information
7	Thread blocks allocated
8	Thread allocation threshold
37	Thread hash buckets
Thread status blocks:
	process/thread 1475/3077674688: active
	process/thread 1318/3078076096: out
	process/thread 1453/3077822144: out
	process/thread 1452/3078510272: out
	process/thread 1274/3078506176: out
	process/thread 1448/3078063808: out
	process/thread 1451/3078309568: out
Environment file handle information
__db.001	file-handle.file name
0	file-handle.mutex [!Set]
0	file-handle.reference count
3	file-handle.file descriptor
0 number
0 size
0 offset
0 count
0 count
0	file-handle.write count
DB_FH_OPENED	file-handle.flags

To fix a corrupted RPM database:

# rm -rf /var/lib/rpm/__db*
# db_verify /var/lib/rpm/Packages
# rpm --rebuilddb

I would also run the following commands to clear out YUMs caches and test the rebuilt RPM database:

# yum clean all
# rpm -qa

Obviously to minimize risk, you should make a backup of all files in /var/lib/rpm/ before attempting to rebuild the RPM database.

What causes a RPM database corruption? It is typically due to an aborted database update process. Occasionally, it occurs when the RPM package itself is updated. When RPM accesses the Berkeley DB files, it makes temporary entries within the tables while it searches for data. If you abort RPM often, this issue will occur much sooner because locks were not cleared when a thread died in Berkeley DB library routines.

The db_verify utility is a Berkeley DB utility which verifies the structure of one or more files and the databases they contain. It should only be used on files that are not being modified by another thread of control. It is used here to ensure that the installed package headers are not corrupted.

Finally, rpm –rebuilddb is used to rebuild the database indices from the installed package headers.

Comments are closed.