среда, 21 сентября 2011 г.

Cassandra migration from 0.6 to 0.7

Cassandra is not mature. I discovered data corruption errors in 0.6. I found nothing that could help me to fix this and I decided to migrate to 0.7 hoping the errors are fixed there.

All you have to do is to follow NEWS file instructions on migration. But there are three pitfalls:

libjna problem in DEB package. Ubuntu has earlier version than required by Cassandra. But package installs fine (wrong dependency version numbers). This results to very strange effects and errors. To fix this you want to install libjna manually, as described here.

Saved caches problem. Before starting up 0.7 you have to manually delete old saved_caches dir. Otherwise you get "Negative array size" exceptions on start up.

Java heap size problem. After fixing previous problems, I discovered a performance degrade in production. Analyzing this I noticed that Java process occupies 13 GB (of 24) RAM. With 0.6 it took about 1-2 GB. In 0.7 Cassandra init scripts set both minimal and maximal (-Xms - Xmx) Java heap sizes to RAM/2. While it is ok for maximum, setting -Xms to 12 GB means that this memory is not going to be used for your actual data. Cassandra accesses data via mmap, and mmap only accesses data in system page cache. Which is shrinked by 12 GB (Java heap). You have to fix manually /etc/cassandra/cassandra-env.sh and set heap to 2 GB (or so).