rm -f /dev/null

Openvswitch with Virtualbox

2012-01-06T07:41:00.000-08:00

This tutorial assume that you already configured openvswitch on virtualbox host machine.

Create the virtual switch on the host machine:

ovs-vsctl add-br lan0

Now we have a virtual switch called lan0 :)

Next we create tap devices on the host machine so we can bind them to the guests laters.

for tap in `seq 0 15`; do
        ip tuntap add mode tap lan0p$tap
done;
# check if the interfaces are there
ip tuntap list

Bring the tap interfaces up:

for tap in `seq 0 15`; do 
        ip link set lan0p$tap up
done;

# check if the interface are up
ip link

Now we use ovs-vsctl to "bind" the tap devices to "lan0" switch

for tap in `seq 0 15`; do
        ovs-vsctl add-port lan0 lan0p$tap
done;

# see that the tap devices are mapped to ports in lan0
ovs-vsctl list-ports lan0

Now we have 16 ports switch called lan0 with 16 tap interfaces binded to lan0 virtual switch.

Next thing is to connect virtualbox guests to it:

I assume you are using the GUI for this so under the network setting select in "Attached to" "Bridge Adapter" and in "Name" select "lan0p1" for port number 1 in the switch.

Repeat this action for each guest you want to connect to the switch.

Now your virtual machines has a virtual network ;)

Enjoy !

Golan

Installing Dena's HandlerSocket NoSQL plugin for MySQL on Centos

2010-10-23T07:43:00.000-07:00

So after reading Yoshinori Matsunobu's blog post I could not resist installing HandlerSocket NoSQL plugin for MySQL. I took a testing machine DELL R510 with 32GB ram, 2x143GB system disks and 12x300GB raid 10 array for MySQL data dir.

So let's start...

Running as root user

cd
mkdir install
cd install

MySQL data dir LVM setup

yum install kmod-xfs.x86_64 xfsprogs.x86_64
pvcreate /dev/sdb
vgcreate mysql /dev/sdb
lvcreate -L500G -n data mysql
mkfs.xfs /dev/mysql/data
echo "/dev/mapper/mysql-data  /var/lib/mysql          xfs     rw,noatime,nobarrier    0 0" >> /etc/fstab
mount /var/lib/mysql

Handler socket source from http://github.com/ahiguti/HandlerSocket-Plugin-for-MySQL

wget http://download.github.com/ahiguti-HandlerSocket-Plugin-for-MySQL-7aceb31.tar.gz

* notte the file name you might get a different one

Downloading MySQL binaries

wget http://mysql_mirror/mysql/Downloads/MySQL-5.1/MySQL-server-community-5.1.51-1.rhel5.x86_64.rpm
wget http://mysql_mirror/mysql/Downloads/MySQL-5.1/MySQL-client-community-5.1.51-1.rhel5.x86_64.rpm
wget http://mysql_mirror/mysql/Downloads/MySQL-5.1/MySQL-shared-compat-5.1.51-1.rhel5.x86_64.rpm
wget http://mysql_mirror/mysql/Downloads/MySQL-5.1/MySQL-devel-community-5.1.51-1.rhel5.x86_64.rpm

Downloading MySQL source

wget http://mysql_mirror/mysql/Downloads/MySQL-5.1/mysql-5.1.51.tar.gz

Installing centos packages

yum install perl libtool gcc make openssl-devel gcc-c++

Installing HandlerSocket

tar zxvf ahiguti-HandlerSocket-Plugin-for-MySQL-7aceb31.tar.gz
cd ahiguti-HandlerSocket-Plugin-for-MySQL-7aceb31
./autogen.sh
./configure --with-mysql-source=/root/install/mysql-5.1.51 --with-mysql-bindir=/usr/bin
make
make install

yum install perl-DBI perl-DBD-MySQL.x86_64
cd perl-Net-HandlerSocket/
perl Makefile.PL
make
make install

Preparing MySQL

cd
cd install
rpm -i MySQL-server-community-5.1.51-1.rhel5.x86_64.rpm
rpm -i MySQL-client-community-5.1.51-1.rhel5.x86_64.rpm
rpm -i MySQL-shared-compat-5.1.51-1.rhel5.x86_64.rpm
rpm -i MySQL-devel-community-5.1.51-1.rhel5.x86_64.rpm

I am using a bit diffrent directory structure for MySQL data dir and logs.
This my.cnf file

[all]
socket=/mysql/mysql.sock

[/usr/bin/perl]
socket=/mysql/mysql.sock

[mysqladmin]
socket=/mysql/mysql.sock

[mysqlslap]
socket=/mysql/mysql.sock

[mysqldump]
socket=/mysql/mysql.sock
quick
max_allowed_packet = 16M

[mysql]
socket=/mysql/mysql.sock
#ervero-auto-rehash
character-set=utf8

[isamchk]
datadir=/var/lib/mysql
key_buffer = 8M
sort_buffer_size = 512M
read_buffer = 2M
write_buffer = 8M

[myisamchk]
datadir=/var/lib/mysql
key_buffer = 512M
sort_buffer_size = 512M
read_buffer = 8M
write_buffer = 8M

[mysqlhotcopy]
datadir=/var/lib/mysql
socket=/mysql/mysql.sock
interactive-timeout

[mysql_installdb]
datadir=/var/lib/mysql

[safe_mysqld]
datadir=/var/lib/mysql
log-error=/mysql/logs/error.log
pid-file=/mysql/mysqld.pid
open-files-limit = 8192

[mysqld]
plugin-load=handlersocket.so
loose_handlersocket_port = 9998
    # the port number to bind to (for read requests)
loose_handlersocket_port_wr = 9999
    # the port number to bind to (for write requests)
loose_handlersocket_threads = 16
    # the number of worker threads (for read requests)
loose_handlersocket_threads_wr = 1
    # the number of worker threads (for write requests)
#open_files_limit = 65535
    # to allow handlersocket accept many concurrent
    # connections, make open_files_limit as large as
    # possible.

auto-increment-increment = 4
auto-increment-offset = 2
character-set-server=utf8
default_table_type = InnoDB
pid-file=/mysql/mysqld.pid
datadir=/var/lib/mysql
socket=/mysql/mysql.sock
log-bin=/mysql/binlogs/handler-socket-test-server-binlog
expire_logs_days=1
relay-log=/mysql/relaylogs/handler-socket-test-server-relay-bin
relay-log-index=/mysql/relaylogs/handler-socket-test-server-relay-bin.index
relay-log-info-file=/mysql/relaylogs/handler-socket-test-server-relay-log.info
master-info-file=/mysql/mysql-master.info
log-slave-updates
server-id=2
slow_query_log=/mysql/logs/slow.log
long_query_time=2
log_short_format
skip-external-locking
skip-name-resolve
max_connections=2000
key_buffer_size = 32M
table_cache=4096
query_cache_type = 0
query_cache_limit = 4M
query_cache_size = 0
long_query_time=3
max_allowed_packet = 16M
sort_buffer_size = 128k
join_buffer_size = 2M
read_buffer_size = 128k
read_rnd_buffer_size = 128k
myisam_sort_buffer_size = 16M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
myisam_recover

innodb_file_per_table
innodb_additional_mem_pool_size = 16M
innodb_buffer_pool_size = 25G
innodb_data_file_path = ibdata1:10M:autoextend
innodb_file_io_threads = 16
innodb_thread_concurrency = 8
innodb_flush_log_at_trx_commit = 0
innodb_log_buffer_size = 8M
innodb_log_file_size = 256M
innodb_log_files_in_group = 3
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 120
innodb_flush_method=O_DIRECT
innodb_commit_concurrency=0

thread_cache_size = 16
thread_cache = 8
thread_concurrency = 8

max_connect_errors = 1000
back_log = 50
thread_stack = 128K
transaction_isolation = REPEATABLE-READ
max_heap_table_size = 256M
tmp_table_size = 256M
bulk_insert_buffer_size = 32M
ft_min_word_len = 4

Now let's configure and start MySQL

mkdir /mysql/{logs,relaylogs,binlogs} -p
chown mysql:mysql /mysql/ -R

rm -rf /var/lib/mysql/*
chown mysql:mysql /var/lib/mysql/
su -c mysql_install_db - mysql
/etc/init.d/mysql start

mysql
show processlist
+----+-------------+-----------------+---------------+---------+------+-------------------------------------------+------------------+
| Id | User        | Host            | db            | Command | Time | State                                     | Info             |
+----+-------------+-----------------+---------------+---------+------+-------------------------------------------+------------------+
|  1 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  2 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  3 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  4 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  5 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  6 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  7 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  8 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  9 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 10 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 11 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 12 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 13 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 14 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 15 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 16 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 17 | system user | connecting host | handlersocket | Connect | NULL | handlersocket: mode=wr, 0 conns, 0 active | NULL             |
| 20 | root        | localhost       | NULL          | Query   |    0 | NULL                                      | show processlist |
+----+-------------+-----------------+---------------+---------+------+-------------------------------------------+------------------+

If you are not seeing HandlerSocket threads in processlist like above something went wrong :(

Create a test table

use test
CREATE TABLE user (
  user_id INT UNSIGNED PRIMARY KEY,
  user_name VARCHAR(50),
  user_email VARCHAR(255),
  created DATETIME 
) ENGINE=InnoDB; 
insert into user (user_id, user_name, user_email, created ) values ( 101, 'golan zakai', 'golan.zakai@lalal4nd.org', now() );

Here is a script for testing copied from Yoshinori Matsunobu's blog post

#!/usr/bin/perl

use strict;
use warnings;
use Net::HandlerSocket;

#1. establishing a connection
my $args = { host => 'ip_to_remote_host', port => 9998 };
my $hs = new Net::HandlerSocket($args);

#2. initializing an index so that we can use in main logics.
 # MySQL tables will be opened here (if not opened)
my $res = $hs->open_index(0, 'test', 'user', 'PRIMARY',
    'user_name,user_email,created');
die $hs->get_error() if $res != 0;

#3. main logic
 #fetching rows by id
 #execute_single (index id, cond, cond value, max rows, offset)
$res = $hs->execute_single(0, '=', [ '101' ], 1, 0);
die $hs->get_error() if $res->[0] != 0;
shift(@$res);
for (my $row = 0; $row < 1; ++$row) {
  my $user_name= $res->[$row + 0];
  my $user_email= $res->[$row + 1];
  my $created= $res->[$row + 2];
  print "$user_name\t$user_email\t$created\n";
}

#4. closing the connection
$hs->close();

Testing...

root@localhost.localdomain ~ <$> perl sample.pl
golan zakai     golan.zakai@lalal4nd.org        2010-10-23 15:07:16
root@localhost.localdomain ~ <$>

The weekend almost over I hope to get a chance to benchmark this soon enough....

Cheers !

Julian's NFCT patch on CentOS

2010-07-12T08:49:00.000-07:00

Recently I had to add Netfilter connection tracking support for IPVS to my LVS directors, after reading:
http://www.ssi.bg/~ja/nfct/HOWTO.txt
http://wiki.centos.org/HowTos/Custom_Kernel

After few long hours of failures I came up with a small build script for this purpose only, for all of you that trying to patch Centos with Julian's NFCT patch I pasted the script here:

SCRIPT

ENJOY !

Golan :)

2008-11-10T13:17:00.000-08:00

Memcached Replication and Namespaces using MySQL Infrastructure.

I was asked to create redundant memcached cluster for various application modules at my company, from developer point of view what I needed most was the option to perform flush operation on memcached from one module without disturbing the other modules running on the same memcached so i figured out memcached namespace capabilities was the thing I needed.

Our application is using memcached heavily for various caching, one of the system requirement was to create a redundant setup, so if one machine is failing you wont lose any keys stored in cache.

Another system requirement was to replicate the data across data centers so local apaches will read only from local cache.

When i came across memcached UDF for mysql I tought combining it with mysql blackhole replication to provide both namespace support and memcached replication can do the trick.

Here i will show one data center cluster setup using four servers two masters and two slaves, The masters are configured with linux-ha LVS abilities (heartbeat and ldirector) and mysql 5.1 while the slaves are configured with memcached, MySQL 5.1, libmemcached and MySQL UDF for memcached.

Masters Roles

Master A will act as writer role for MySQL and will forward all data to Master B and all slaves via MySQL replication binlog.

Master B will act as memcache reader role and will preform load balancing between the memcached servers.Theoretically Master B can be replaced with hardware load balancer and function only as a backup for Master A.

In this setup in case Master A is failing Master B will take his place and vice verse.

Slaves Roles

Read Master A binarylog and write them to memcached via MySQL UDF for memcache.

Answer memcache get request from Master B.

Installation

All the setup was preformed on centos 5.2 so try to install the needed software as your distro recommend.
On both masters you will to install heartbeat and ldirector from linux-ha.

yum install heartbeat.x86_64 heartbeat-devel.x86_64 heartbeat-ldirectord.x86_64 heartbeat-devel.x86_64
* note architecture

In this setup both master have 2 network interfaces each network interface is facing different network network A is where the application lives and network B is where the memcached cluster lives.
network A 10.100.5.0/24
network B 10.200.5.0/24

master A and B are acting as routers between the 2 networks.

network settings

Master A
eth0 10.100.5.10
eth1 10.200.5.10

Master B
eth0 10.100.5.20
eth1 10.200.5.20

Slave 1
eth0 10.200.5.100

Slave 2
eth0 10.200.5.101

Virtual IP (VIP) configuration haresources file at /etc/ha.d/

master-a.yourdomain.com IPaddr2::10.100.5.2/24/eth0 IPaddr2::10.200.5.2/24/eth1 mysql-takeover
master-b.yourdomain.com IPaddr2::10.100.5.3/24/eth0 IPaddr2::10.200.5.3/24/eth1 ldirectord::ldirectord-memcached.cf

Load balancer configuration ldirectord-memcached.cf at /etc/ha.d/

checktimeout=30
checkinterval=10
autoreload=yes
logfile="/var/log/ldirectord.log"
emailalert="golanzakai@yourdomain.com"
emailalertfreq=3600
emailalertstatus=all
quiescent=yes

virtual=10.100.5.3:11211
real=10.200.5.100:11211 masq
real=10.200.5.101:11211 masq
scheduler=rr
netmask=255.255.255.255
protocol=tcp
checktype=ping

Slaves IP Addresses file at /etc/slaves-ip.txt

10.200.5.100
10.200.5.101

MySQL take over resource file at /etc/ha.d/resource.d/mysql-takeover

MYSQL_VIP=10.200.5.2

MYSQL_MASTER_USER=master
MYSQL_MASTER_PASS=master

MYSQL_SLAVE_USER=slave
MYSQL_SLAVE_PASS=slave

LOGFILE=/tmp/mysql-takeover-log

start() {

CHANGE_MASTER=`echo "show master status;" | mysql | tail -n 1 | awk -F " " -v q=\' '{print " \
CHANGE MASTER TO \
MASTER_HOST="q"MYSQL_REPLICATION_HOST"q", \
MASTER_USER="q"MYSQL_REPLICATION_USER"q", \
MASTER_PASSWORD="q"MYSQL_REPLICATION_PASS"q", \
MASTER_LOG_FILE="q$1q",MASTER_LOG_POS="$2";"}' \
| sed -e 's/MYSQL_REPLICATION_HOST/'$MYSQL_VIP'/' \
| sed -e 's/MYSQL_REPLICATION_USER/'$MYSQL_MASTER_USER'/' \
| sed -e 's/MYSQL_REPLICATION_PASS/'$MYSQL_MASTER_PASS'/'`

echo "slave stop;$CHANGE_MASTER;slave start;" >> $LOGFILE

for MYSQL_IP in `cat /etc/slaves-ip.txt`; do
echo "sending master string to " $MYSQL_IP >> $LOGFILE
echo "mysql -h $MYSQL_IP -u${MYSQL_SLAVE_USER} -p${MYSQL_SLAVE_PASS}" >> $LOGFILE
echo "slave stop;$CHANGE_MASTER;slave start;" | mysql -h $MYSQL_IP -u${MYSQL_SLAVE_USER} -p${MYSQL_SLAVE_PASS}
done
}
stop() {
echo "mysql-takeover stop" >> $LOGFILE
}
status() {
echo "mysql-takeover status" >> $LOGFILE
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status
;;
*)
echo "mysql-takeover no params" >> $LOGFILE
;;
esac

For each machine in the cluster you will install mysql 5.1.

download mysql 5.1 from http://dev.mysql.com/downloads/mysql/5.1.html for your server architecture.

rpm -i MySQL-devel-community-5.1.29-0.rhel5.x86_64.rpm
rpm -i MySQL-server-community-5.1.29-0.rhel5.x86_64.rpm
rpm -i MySQL-client-community-5.1.29-0.rhel5.x86_64.rpm
rpm -i MySQL-shared-community-5.1.29-0.rhel5.x86_64.rpm

I install it like that because i couldn't find an up to date repository.

make sure you have those lines at my.cf

Master A
[mysqld]

log-bin=mysql-bin
log-slave-updates

replicate-do-db=memcached
server-id=50

Master B
[mysqld]

log-bin=mysql-bin
log-slave-updates

replicate-do-db=memcached
server-id=51

here is the time to tune other mysql variables as you wish.

restart mysql on both masters.

on master A mysql console paste this:

drop database IF EXISTS memcached;
create database memcached;
use memcached;

drop table IF EXISTS mc_namespaces;
drop table IF EXISTS mc_stored_keys;
drop table IF EXISTS mc_storage;

create table mc_namespaces( ns_id int, ns_name varchar(16) ) ENGINE=InnoDB;
create table mc_stored_keys( ns_id int, mc_key char(32), created timestamp, expire int ) ENGINE=InnoDB;
create table mc_storage( ns_id int, mc_key char(32), mc_value mediumtext) ENGINE=BLACKHOLE;

drop PROCEDURE IF EXISTS set_key;

delimiter $$

CREATE PROCEDURE set_key( IN f_ns_id int, IN f_mc_key CHAR(32), IN f_mc_value mediumtext, IN f_expire int )
BEGIN
INSERT INTO mc_stored_keys( ns_id, mc_key, created, expire ) values ( f_ns_id, f_mc_key, CURRENT_TIMESTAMP, f_expire );
INSERT INTO mc_storage( ns_id, mc_key, mc_value ) values ( f_ns_id, f_mc_key, f_mc_value );
END $$

now test your configuration by creating 1 namespace and setting 2 keys:

insert into mc_namespaces( ns_id, ns_name ) values ( 1, 'my_namespace' );
call set_key( 1, '12345678901234567890123456789012', 'l4l4nd', 100 );
call set_key( 1, '12345678901234567890123456789ABC', 'w4st3l4nd', 100 );

/etc/init.d/heartbeat start;

login to master B and see if it replicate the data on tables mc_namespaces and mc_stored_keys.
if not something went wrong in the replication setup or your heartbeat ldirector configuration is not okay.

Slaves setup

download libmemcached and MySQL UDF for memcache from http://tangent.org/ i tried UDF version 0.6 and it had a bug so i installed 0.5 and it was working fine, I have noticed 0.7 is released but i didnt try it yet while i am pretty sure patrick fixed the bug.

make sure at this point to have memcached up and running on localhost.

yum install automake
yum install pkgconfig
wget http://download.tangent.org/memcached_functions_mysql-0.5.tar.gz
tar zxvf memcached_functions_mysql-0.5.tar.gz
cd memcached_functions_mysql-0.5
# locate your architecture paths
./configure --with-mysql=/usr/bin/mysql_config --libdir=/usr/lib64/mysql/plugin/
make
make install
mysql < ./sql/install_functions.sql

edit your my.cf file and add this line under [mysqld]:

replicate-do-table=memcached.mc_storage

restart mysql

on slave mysql console:

select memc_servers_set('localhost:11211');

CREATE TABLE `mc_storage` (
`ns_id` int(11) DEFAULT NULL,
`mc_key` char(32) DEFAULT NULL,
`mc_value` mediumtext
) ENGINE=BLACKHOLE DEFAULT CHARSET=utf8;

CREATE TRIGGER key_storage
BEFORE INSERT ON mc_storage
FOR EACH ROW BEGIN
set @mm = memc_set(NEW.mc_key, NEW.mc_value);
END

repeat this for each slave you want to install.

now restart heartbeat on master A and master B make sure they have taken thier resources.

now check if it's working:

on master A mysql console do this:

use memcached

call set_key( 1, '12345678901234567890123456789ABC', 'w4st3l4nd', 100 );

On both slaves shell do this:

memcat --servers localhost 12345678901234567890123456789ABC

if you see w4st3l4nd your slave have writen the data to memcache

Now let's test the get request from the Network A via the Load balancer IP login to 1 of you application servers on network a and type:

memcat --servers 10.100.5.20 12345678901234567890123456789ABC

If you get the value your setup is completed.

This is only proof of concept and here i showen only set_key function but implementing the rest of them with nice triggers is easy.

Have Fun.