Memcached Replication and Namespaces using MySQL Infrastructure.

I was asked to create redundant memcached cluster for various application modules at my company, from developer point of view what I needed most was the option to perform flush operation on memcached from one module without disturbing the other modules running on the same memcached so i figured out memcached namespace capabilities was the thing I needed.

Our application is using memcached heavily for various caching, one of the system requirement was to create a redundant setup, so if one machine is failing you wont lose any keys stored in cache.

Another system requirement was to replicate the data across data centers so local apaches will read only from local cache.

When i came across memcached UDF for mysql I tought combining it with mysql blackhole replication to provide both namespace support and memcached replication can do the trick.

Here i will show one data center cluster setup using four servers two masters and two slaves, The masters are configured with linux-ha LVS abilities (heartbeat and ldirector) and mysql 5.1 while the slaves are configured with memcached, MySQL 5.1, libmemcached and MySQL UDF for memcached.

Masters Roles

Master A will act as writer role for MySQL and will forward all data to Master B and all slaves via MySQL replication binlog.

Master B will act as memcache reader role and will preform load balancing between the memcached servers.Theoretically Master B can be replaced with hardware load balancer and function only as a backup for Master A.

In this setup in case Master A is failing Master B will take his place and vice verse.

Slaves Roles

Read Master A binarylog and write them to memcached via MySQL UDF for memcache.

Answer memcache get request from Master B.

---

---

---

Installation

All the setup was preformed on centos 5.2 so try to install the needed software as your distro recommend.
On both masters you will to install heartbeat and ldirector from linux-ha.


yum install heartbeat.x86_64 heartbeat-devel.x86_64 heartbeat-ldirectord.x86_64 heartbeat-devel.x86_64
* note architecture

In this setup both master have 2 network interfaces each network interface is facing different network network A is where the application lives and network B is where the memcached cluster lives.
network A 10.100.5.0/24
network B 10.200.5.0/24

master A and B are acting as routers between the 2 networks.

network settings

Master A
eth0 10.100.5.10
eth1 10.200.5.10

Master B
eth0 10.100.5.20
eth1 10.200.5.20

Slave 1
eth0 10.200.5.100

Slave 2
eth0 10.200.5.101

Virtual IP (VIP) configuration haresources file at /etc/ha.d/

master-a.yourdomain.com IPaddr2::10.100.5.2/24/eth0 IPaddr2::10.200.5.2/24/eth1 mysql-takeover
master-b.yourdomain.com IPaddr2::10.100.5.3/24/eth0 IPaddr2::10.200.5.3/24/eth1 ldirectord::ldirectord-memcached.cf

Load balancer configuration ldirectord-memcached.cf at /etc/ha.d/

checktimeout=30
checkinterval=10
autoreload=yes
logfile="/var/log/ldirectord.log"
emailalert="golanzakai@yourdomain.com"
emailalertfreq=3600
emailalertstatus=all
quiescent=yes

virtual=10.100.5.3:11211
real=10.200.5.100:11211 masq
real=10.200.5.101:11211 masq
scheduler=rr
netmask=255.255.255.255
protocol=tcp
checktype=ping

Slaves IP Addresses file at /etc/slaves-ip.txt

10.200.5.100
10.200.5.101

MySQL take over resource file at /etc/ha.d/resource.d/mysql-takeover

MYSQL_VIP=10.200.5.2

MYSQL_MASTER_USER=master
MYSQL_MASTER_PASS=master

MYSQL_SLAVE_USER=slave
MYSQL_SLAVE_PASS=slave

LOGFILE=/tmp/mysql-takeover-log

start() {

CHANGE_MASTER=`echo "show master status;" | mysql | tail -n 1 | awk -F " " -v q=\' '{print " \
CHANGE MASTER TO \
MASTER_HOST="q"MYSQL_REPLICATION_HOST"q", \
MASTER_USER="q"MYSQL_REPLICATION_USER"q", \
MASTER_PASSWORD="q"MYSQL_REPLICATION_PASS"q", \
MASTER_LOG_FILE="q$1q",MASTER_LOG_POS="$2";"}' \
| sed -e 's/MYSQL_REPLICATION_HOST/'$MYSQL_VIP'/' \
| sed -e 's/MYSQL_REPLICATION_USER/'$MYSQL_MASTER_USER'/' \
| sed -e 's/MYSQL_REPLICATION_PASS/'$MYSQL_MASTER_PASS'/'`

echo "slave stop;$CHANGE_MASTER;slave start;" >> $LOGFILE

for MYSQL_IP in `cat /etc/slaves-ip.txt`; do
echo "sending master string to " $MYSQL_IP >> $LOGFILE
echo "mysql -h $MYSQL_IP -u${MYSQL_SLAVE_USER} -p${MYSQL_SLAVE_PASS}" >> $LOGFILE
echo "slave stop;$CHANGE_MASTER;slave start;" | mysql -h $MYSQL_IP -u${MYSQL_SLAVE_USER} -p${MYSQL_SLAVE_PASS}
done
}
stop() {
echo "mysql-takeover stop" >> $LOGFILE
}
status() {
echo "mysql-takeover status" >> $LOGFILE
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status
;;
*)
echo "mysql-takeover no params" >> $LOGFILE
;;
esac

For each machine in the cluster you will install mysql 5.1.

download mysql 5.1 from http://dev.mysql.com/downloads/mysql/5.1.html for your server architecture.

rpm -i MySQL-devel-community-5.1.29-0.rhel5.x86_64.rpm
rpm -i MySQL-server-community-5.1.29-0.rhel5.x86_64.rpm
rpm -i MySQL-client-community-5.1.29-0.rhel5.x86_64.rpm
rpm -i MySQL-shared-community-5.1.29-0.rhel5.x86_64.rpm

I install it like that because i couldn't find an up to date repository.

make sure you have those lines at my.cf

Master A
[mysqld]

log-bin=mysql-bin
log-slave-updates

replicate-do-db=memcached
server-id=50

Master B
[mysqld]

log-bin=mysql-bin
log-slave-updates

replicate-do-db=memcached
server-id=51

here is the time to tune other mysql variables as you wish.

restart mysql on both masters.


on master A mysql console paste this:

drop database IF EXISTS memcached;
create database memcached;
use memcached;

drop table IF EXISTS mc_namespaces;
drop table IF EXISTS mc_stored_keys;
drop table IF EXISTS mc_storage;

create table mc_namespaces( ns_id int, ns_name varchar(16) ) ENGINE=InnoDB;
create table mc_stored_keys( ns_id int, mc_key char(32), created timestamp, expire int ) ENGINE=InnoDB;
create table mc_storage( ns_id int, mc_key char(32), mc_value mediumtext) ENGINE=BLACKHOLE;

drop PROCEDURE IF EXISTS set_key;

delimiter $$

CREATE PROCEDURE set_key( IN f_ns_id int, IN f_mc_key CHAR(32), IN f_mc_value mediumtext, IN f_expire int )
BEGIN
INSERT INTO mc_stored_keys( ns_id, mc_key, created, expire ) values ( f_ns_id, f_mc_key, CURRENT_TIMESTAMP, f_expire );
INSERT INTO mc_storage( ns_id, mc_key, mc_value ) values ( f_ns_id, f_mc_key, f_mc_value );
END $$

now test your configuration by creating 1 namespace and setting 2 keys:

insert into mc_namespaces( ns_id, ns_name ) values ( 1, 'my_namespace' );
call set_key( 1, '12345678901234567890123456789012', 'l4l4nd', 100 );
call set_key( 1, '12345678901234567890123456789ABC', 'w4st3l4nd', 100 );

/etc/init.d/heartbeat start;

login to master B and see if it replicate the data on tables mc_namespaces and mc_stored_keys.
if not something went wrong in the replication setup or your heartbeat ldirector configuration is not okay.

Slaves setup

download libmemcached and MySQL UDF for memcache from http://tangent.org/ i tried UDF version 0.6 and it had a bug so i installed 0.5 and it was working fine, I have noticed 0.7 is released but i didnt try it yet while i am pretty sure patrick fixed the bug.

make sure at this point to have memcached up and running on localhost.

yum install automake
yum install pkgconfig
wget http://download.tangent.org/memcached_functions_mysql-0.5.tar.gz
tar zxvf memcached_functions_mysql-0.5.tar.gz
cd memcached_functions_mysql-0.5
# locate your architecture paths
./configure --with-mysql=/usr/bin/mysql_config --libdir=/usr/lib64/mysql/plugin/
make
make install
mysql < ./sql/install_functions.sql

edit your my.cf file and add this line under [mysqld]:

replicate-do-table=memcached.mc_storage

restart mysql

on slave mysql console:

select memc_servers_set('localhost:11211');

CREATE TABLE `mc_storage` (
`ns_id` int(11) DEFAULT NULL,
`mc_key` char(32) DEFAULT NULL,
`mc_value` mediumtext
) ENGINE=BLACKHOLE DEFAULT CHARSET=utf8;

CREATE TRIGGER key_storage
BEFORE INSERT ON mc_storage
FOR EACH ROW BEGIN
set @mm = memc_set(NEW.mc_key, NEW.mc_value);
END

repeat this for each slave you want to install.

now restart heartbeat on master A and master B make sure they have taken thier resources.

now check if it's working:

on master A mysql console do this:

use memcached

call set_key( 1, '12345678901234567890123456789ABC', 'w4st3l4nd', 100 );

On both slaves shell do this:

memcat --servers localhost 12345678901234567890123456789ABC

if you see w4st3l4nd your slave have writen the data to memcache

Now let's test the get request from the Network A via the Load balancer IP login to 1 of you application servers on network a and type:

memcat --servers 10.100.5.20 12345678901234567890123456789ABC

If you get the value your setup is completed.

This is only proof of concept and here i showen only set_key function but implementing the rest of them with nice triggers is easy.

Have Fun.