Cassandra migration from 0.6 to 0.7

Cassandra is not mature. I discovered data corruption errors in 0.6. I found nothing that could help me to fix this and I decided to migrate to 0.7 hoping the errors are fixed there.

All you have to do is to follow NEWS file instructions on migration. But there are three pitfalls:

libjna problem in DEB package. Ubuntu has earlier version than required by Cassandra. But package installs fine (wrong dependency version numbers). This results to very strange effects and errors. To fix this you want to install libjna manually, as described here.

Saved caches problem. Before starting up 0.7 you have to manually delete old saved_caches dir. Otherwise you get "Negative array size" exceptions on start up.

Java heap size problem. After fixing previous problems, I discovered a performance degrade in production. Analyzing this I noticed that Java process occupies 13 GB (of 24) RAM. With 0.6 it took about 1-2 GB. In 0.7 Cassandra init scripts set both minimal and maximal (-Xms - Xmx) Java heap sizes to RAM/2. While it is ok for maximum, setting -Xms to 12 GB means that this memory is not going to be used for your actual data. Cassandra accesses data via mmap, and mmap only accesses data in system page cache. Which is shrinked by 12 GB (Java heap). You have to fix manually /etc/cassandra/cassandra-env.sh and set heap to 2 GB (or so).

Apache Cassandra experience

At one of my projects I switched from Postgresql to Cassandra. There were reasons for the switch.

First. For each user I had to keep an inbox for storing incoming messages and events. What is inbox? It is a sorted collection of items. Items are accessed using ranged queries. This caused huge IO overhead on Postgres, because of lack of clustered indexes. All "tables" in Cassandra are clustered, because they are kept as SST (sorted string tables).

Second. My application had huge write throughput. Postgres is good at write with all that write-ahead logs and absence of table-locks on write. And even after write-aware optimizations it still was not enough. Cassandra's data write process is completely different. And it better suits my needs.

Third. Application servers are Python Twisted applications. There is one Postgres binding for Twisted and it is abandoned and buggy. Cassandra API is available via Thrift, which in turn supports Twisted. I recommend great Telephus wrapper for Thrift and Twisted.

At Cassandra's IRC channel people are telling each other of their Cassandra clusters. I look a bit stupid when saying I have a single node. But who cares? If it works better than Postgres for me - why not?

Disclaimer: I am not telling here that Cassandra is better than Postgresql. It just suits better in this certain application. I use Postgresql a lot at in many other projects.