Alternative Redis-Like Databases with Python
Redis is one of the more unique NoSQL offerings to have become popular over the past five years. It seems that there is no limit to the use-cases one can find for Redis. It's fantastic as a cache, doubles as a task-queue, can provide fast type-ahead search, and much more. The idea that you can store data-structures instead of rows and columns, keys and values, or JSON documents strikes me as particularly innovative. A while back I released walrus, a collection of Python utilities I'd built to simplify some of these use-cases and provide Pythonic APIs for the data-structures Redis natively supports. If you're a Python developer you might check it out.
Recently I've learned about a few new Redis-like databases: Rlite, Vedis and LedisDB. Each of these projects offers a slightly different take on the data-structure server you find in Redis, so I thought that I'd take some time and see how they worked. In this post I'll share what I've learned, and also show you how to use these databases with Walrus, as I've added support for them in the latest 0.3.0 release.
Rlite
Rlite is to Redis what SQLite is to Postgresql. What this means is that Rlite provides virtually all the same functionality of Redis in a serverless, zero-config library you embed as part of your application. Some of Rlite's features are:
- Serverless / zero-config library.
- Support for virtually every Redis command.
- Supports transactions.
- Scripting via Lua.
- Databases can be either in-memory or stored in a single file on disk. By storing data on-disk Rlite allows you to work with data-sets that exceed available memory.
- Lots of language support: objective-c, golang, java, javascript, php, python, ruby, r, and of course C.
Rlite could be used for mobile development, where a separate server process is not practical. Rlite would also work well in development, testing or continuous-integration environments. The project's README suggests using Rlite as a slave of your Redis cluster to provide additional durability. Rlite might also be useful for large data-sets that do not fit in RAM.
The author of Rlite has also written the Python bindings, which can be installed using pip:
$ pip install hirlite
Rlite seems like a very cool project. The compatibility with Redis is a strong selling point since it meant I could re-use a lot of my existing tools. If you're curious about how Rlite stores data on disk, the author has provided documentation about the RLD file format and write-ahead log.
Vedis
Vedis is another embedded database with over 70 Redis-like commands. Like Rlite, Vedis is serverless, zero-config, and supports both in-memory and on-disk databases. According to the project's README,
Vedis is a self-contained C library without dependency. It requires very minimal support from external libraries or from the operating system. This makes it well suited for use in embedded devices that lack the support infrastructure of a desktop computer. This also makes Vedis appropriate for use within applications that need to run without modification on a wide variety of computers of varying configurations.
The distinctive features of Vedis are:
- Serverless / zero-config library.
- Transactional (ACID) datastore.
- Databases can be either in-memory or stored in a single file on disk.
- Built on fast key/value store, which can be used directly for fast key/value operations.
- Cross-platform file format.
- Thread-safe and fully re-entrant.
- Python bindings allow you to write your own commands in Python.
Use-cases for Vedis are similar to Rlite, but Vedis is not nearly as compatible with Redis, so it's kind of it's own thing. I can see it being useful in mobile or resource-constrained environments, as well as for local development and testing. Vedis might also be useful for data-sets that don't fit in RAM.
It's important to note that Vedis supports a more limited subset of commands than Rlite. Specifically, Vedis is entirely lacking a sorted set data-type, and the list implementation only offers a handful of methods (index, push, pop, length). The hash and set data-types, however, are pretty full-featured.
I happen to be the author of the vedis-python bindings, which use ctypes. One neat feature is that you can extend Vedis with your own commands, implemented in Python. The Python bindings can be installed with pip:
$ pip install vedis
Unfortunately, the developers of Vedis, Symisc Systems, have indicated that the project is not currently being actively developed. Symisc is doing some really neat stuff, though, and hopefully they will pick the project back up.
LedisDB
LedisDB is the most Redis-like of the projects discussed in this post. Unlike the previous databases, LedisDB can run as a separate server process or be embedded in a golang app. LedisDB is unique in that it supports a number of different storage backends such as LevelDB, RocksDB, LMDB, and more. It seems to me to have parity with Redis' features.
LedisDB has the following distinctive features:
- Supports virtually every Redis command.
- Supports multiple storage backends, including LevelDB, RocksDB, LMDB, BoltDB, and in-memory databases.
- Size of data-set is not limited by RAM.
- Transactions.
- Supports the Redis protocol for communication, so most clients and other tools work with LedisDB.
- RESTful API allows you to communicate with LedisDB over HTTP.
- Written in golang, easy to deploy.
I see the use-cases for LedisDB to be primarily derived from it's ability to work with multiple key/value storage backends, and as a consequence, to store data-sets that exceed available RAM. The HTTP API is a cool addition, though thanks to it's compatibility with Redis' wire protocol, existing tools will probably work with LedisDB as well.
The python bindings can be installed with pip:
$ pip install ledis
LedisDB seems like a really neat project, and the author provides some nice benchmarks showing Ledis lagging only slightly behind Redis for many use-cases, which is impressive given that it is using on-disk databases.
Using with Walrus
With the version 0.3.0 release, I've added a new tusks submodule to the walrus package that contains classes for working with the above libraries. To use a particular tusk, just make sure the appropriate Python bindings are installed.
The Walrus project is, at the most basic level, simply a wrapper around Andy McCurdy's redis-py library. Walrus provides various high-level APIs on top of this, but that's the gist. If you'd like a quick introduction to Walrus, check out this blog post.
Rlite + Walrus = WalrusLite
To use Rlite with Walrus, you can use the WalrusLite class, found in the walrus.tusks.rlite module.
>>> from walrus.tusks.rlite import WalrusLite
>>> db = WalrusLite(':memory:')
>>> huey = db.Hash('huey')
>>> huey.update(color='white', temperament='ornery', type='kitty')
<Hash "huey": {'color': 'white', 'type': 'kitty', 'temperament': 'ornery'}>
>>> huey.keys()
['color', 'type', 'temperament']
>>> 'color' in huey
True
>>> huey['color']
'white'
At the time of writing, the only important commands that did not seem to be implemented were the HSCAN, SSCAN and ZSCAN commands which are used for efficient iteration. In their place, Walrus provides compatible implementations that rely on HGETALL, SMEMBERS and ZRANGE, respectively.
Vedis + Walrus = WalrusVedis
To use Vedis with Walrus, you can use the WalrusVedis class, found in the walrus.tusks.vedisdb module. In the examples below we'll implement two custom commands: a simple command that converts a string to title-case, and a command to calculate the union of two sets.
>>> from walrus.tusks.vedisdb import WalrusVedis
>>> db = WalrusVedis(':memory:') # Create an in-memory database.
>>> @db.command('SUNION') # Vedis supports SDIFF and SINTER, but not SUNION.
... def sunion(context, key1, key2):
... return list(db.smembers(key1) | db.smembers(key2))
...
>>> @db.command('KTITLE')
... def ktitle(context, key):
... source_val = context[key]
... if source_val:
... return source_val.title()
... return False
...
>>> s1 = db.Set('s1')
>>> s1.add(*range(3))
3
>>> s2 = db.Set('s2')
>>> s2.add(*range(1, 5))
4
>>> db.SUNION('s1', 's2')
['1', '0', '3', '2', '4']
>>> db['name'] = 'charlie leifer'
>>> db.KTITLE('name')
'Charlie Leifer'
LedisDB + Walrus = WalrusLedis
To use LedisDB instead of Redis in your Python application, you can use the WalrusLedis class in place of the usual Walrus object:
>>> from walrus.tusks.ledisdb import WalrusLedis
>>> db = WalrusLedis()
Like Rlite, you should be able to use WalrusLedis just as you would use the normal Walrus object.
Thanks for reading
Thanks for taking the time to read this post, I hope you found it interesting. If you'd like to try out these alternate backends with walrus, check out the alternate backend doc, which contains a condensed version of the material in this post.
Do you know of any other Redis-like databases out there? I'd be very interested to hear about them!
