engine.so

Embeddable data storage engines.

Tracking, Benchmarking and Sharing Information about an open source embedded data storage engines, internals, architectures, data storage and transaction processing.

If you would like to share a material or make a correction, please use GitHub or contact me by Email.

Storage Engines

name language storage description
Berkeley DB C B-Tree, Hash Berkeley DB (BDB) is a software library that provides a high-performance embedded database for key/value data.
Tokyo/Kyoto Cabinet C++ B-Tree, Hash Tokyo Cabinet and Kyoto Cabinet are two libraries of routines for managing key-value databases.
SQLite C B-Tree, LSM SQLite is a relational database management system contained in a C programming library.
UnQLite C Hash UnQLite is a in-process software library which implements a self-contained, serverless, zero-configuration, transactional NoSQL database engine.
LevelDB C++ LSM LevelDB is an open source on-disk key-value store written by Google.
LevelDB by Basho C++ LSM A fork of LevelDB modified to benefit the Riak environment.
LevelDB by HyperDex C++ LSM A fork of LevelDB intended to meet the needs of HyperDex while remaining compatible with LevelDB.
RocksDB C++ LSM RocksDB is an embeddable persistent key-value store for fast storage from Facebook. (LevelDB by Facebook)
Sophia C Sophia Hybrid RAM-Disk Embeddable Transactional Key-Value Storage.
WiredTiger C B-Tree, LSM WiredTiger is a high performance, scalable, production quality, NoSQL, Open Source extensible platform for data management.
Percona/TokuFT C++ Fractal Tree PerconaFT is a high-performance, transactional key-value store.
ForestDB C++ HB+-Trie Forestdb - A Fast Key-Value Storage Engine Based on Hierarchical B+-Tree Trie.
Sparkey C Hash Simple constant key/value storage library, for read-heavy systems with infrequent large bulk inserts.
LMDB C B-Tree Lightning Memory-Mapped Database (LMDB) is a software library that provides a high-performance embedded transactional database in the form of a key-value store.
MDBX C B-Tree Modified version of LMDB (Symas Lightning Memory-Mapped Database)
NessDB C Buffered-Tree A very fast transactional key-value, embedded database storage engine.
Vedis C Hash Vedis is an embeddable datastore C library built with over 70 commands similar in concept to Redis but without the networking layer.
Rlite C Hash Self-contained, serverless, zero-configuration, transactional Redis-compatible database engine.
SkipDB C Skip List Small, portable, ACID, C implemented BDB style database based on skiplists instead of b-trees.
EJDB C B-Tree, Hash Embedded JSON Database engine C library.
UpscaleDB C B-Tree A very fast lightweight embedded database engine with built-in analytic functions.
Eblob C Eblob is an append-only low-level IO library, which saves data in blob files.
WhiteDB C Hash WhiteDB is a lightweight NoSQL database library written in C, operating fully in main memory.

Benchmarking

name description
IOArena IOArena is an utility designed for evaluating performance of embedded databases.

Papers

name year description
Organization and maintenance of large ordered indexes 1972 Organization and maintenance of an index for a dynamic random access file is considered. The pages of the index are organized in a special data-structure, so-called B-trees.
The Log-Structured Merge-Tree (LSM-Tree) 1996 The LSM tree is a data structure with performance characteristics that make it attractive for providing indexed access to files with high insert volume, such as transactional log data.
The Buffer Tree: A Technique for Designing Batched External Data Structures 2003 A technique for designing external memory data structures that support batched operations I/O efficiently.
Stratified B-trees and Versioned Dictionaries 2011 The 'Stratified B-tree' is the first versioned dictionary offering fast updates and an optimal tradeoff between space, query and update costs.
Anti-Caching: A New Approach to Database Management System Architecture 2013 To overcome the restriction that all data fit in main memory, we propose a new technique, called anti-caching, where cold data is moved to disk in a transactionally-safe manner as the database grows in size.
A Comparison of Fractal Trees to Log-Structured Merge (LSM) Trees 2014 This paper explains the advantages of Fractal-Tree (R) indexing compared to Log-Structured Merge (LSM) trees.
Log-structured Memory for DRAM-based Storage 2014 A log-structured approach to memory management allows 80-90% memory utilization while offering high performance.
Rethinking SIMD Vectorization for In-Memory Databases 2015 In this paper, we present novel vectorized designs and implementations of database operators, based on advanced SIMD operations, such as gathers and scatters, study selections, hash tables, and partitioning.
WiscKey: Separating Keys from Values in SSD-conscious Storage 2016 We present WiscKey, a persistent LSM-tree-based key-value store with a performance-oriented data layout that separates keys from values to minimize I/O amplifi-cation.


 Dmitry Simonenko