在大数据的浪潮下,redis只提供主从分布模式越来越显得捉襟见肘了。2015年3月底,Redis-Cluster终于出了正式版本,真可谓千呼万唤始出来啊。下面我们就来看看如何一步一步安装和配置Redis集群。
全文共分9个安装步骤进行介绍,提供了所有安装相关的自动化脚本源码,这些脚本可以通过配置文件扩展到多台机器多个实例的redis-cluster安装。 操作系统:Centos6.4 Redis版本: 3.0 机器(64位):3台机器用于安装redis集群+1台机器做总控 网路环境: 4台机器在同一个局域网 集群拓扑结构:9个实例,3主6从,每个主2个从节点。 1. 首先到redis官网下载redis3.0版本, 程序文件包居然只有1.3MB,赞一个。
wget http://download.redis.io/releases/redis-3.0.0.tar.gz
2. 准备可执行程序:在总控机器上解压&编译
tar xzvf redis-3.0.0.tar.gz
cd redis-3.0.0
make
这个时候在src目录下已经生成了redis的可执行程序了(绿色部分)
3. 准备redis的配置文件: 这里根据:http://redis.io/topics/cluster-tutorial写了个最简单版本的(文件名redis.conf):
port 30000
daemonize yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly no
loglevel warning
logfile ../logs/redis.log
pidfile ../logs/redis.pid
timeout 60
tcp-keepalive 60
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir ../logs/
save 900 1
save 300 10
save 60 100000
maxclients 65536
slave-read-only yes
4. 打包程序和配置文件:使用如下的脚本,打包之后方便拷贝到三台待安装机器上部署(文件名tar_redis.sh).
#!/bin/bash
#打包redis程序和配置
if [ $# -ne 0 ]
then
echo "Usage: $0"
exit 1
fi
rm -rf redis-server.tar.gz
mkdir -p redis-server
mkdir -p redis-server/conf
mkdir -p redis-server/sbin
mkdir -p redis-server/logs
cp -f ./redis.conf redis-server/conf/
cd ./redis-3.0.0/src/
cp -f mkreleasehdr.sh redis-benchmark redis-check-aof redis-check-dump redis-cli \
redis-sentinel redis-server redis-trib.rb ../../redis-server/sbin/
cd -
tar czvf redis-server.tar.gz redis-server
exit 0
sh tar_redis.sh执行后生成程序和配置文件包redis-server.tar.gz
5. 分发程序和配置文件包到3台待安装安装的机器, 这里我们也写了个自动脚本dist_redis.sh. (使用这个脚本需要先安装sshpass做远程自动登录[见另一篇博文Linux自动密码登陆利器sshpass中的安装和问题说明], sshpass是建立信任关系之外的另一种选择).该脚本的功能是:
- 删除就的备份目录并备份当前目录
- 拷贝程序包到制定目录 并 解压
- 设置配置文件中的端口
- 执行方法: sh dist_redis.sh ipportlist.txt
#!/bin/bash
if [ $# -ne 1 ]
then
echo "Usage: $0 iplistFile"
exit 1
fi
#自动分发redis程序和配置到所有机器[见输入的iplistFile]
username='root'
password='123456'
dist_dir=/home/redis/redis-server/
exe_files="./redis-server.tar.gz"
echo ${dist_dir}
echo ${exe_files}
while read ipport
do
ip=`echo ${ipport} |cut -d: -f1`
port=`echo ${ipport} | cut -d: -f2`
curr_dir="`echo ${dist_dir}|sed 's:/$::g'`-${port}"
bak_dir=`echo "${curr_dir}" | sed 's:/$::g'`_bak
echo "process host:${ip}:${port} at ${curr_dir}============================================="
echo "step 1: rm old bak:${bak_dir}------------------------------------------"
sshpass -p ${password} ssh ${username}@${ip} "rm -rf ${bak_dir} 0</dev/null" 0</dev/null
echo $?
echo "step 2: backup data-------------------------------------"
sshpass -p ${password} ssh ${username}@${ip} "mv -f ${curr_dir} ${bak_dir} 2>/dev/null 1>&2 0</dev/null" 0</dev/null
echo $?
echo "step 3: mkdir dir:${curr_dir}---------------------------"
sshpass -p ${password} ssh ${username}@${ip} "mkdir -p ${curr_dir} 0</dev/null" 0</dev/null
echo $?
echo "step 4: copy tarfiles------------------------------------"
sshpass -p ${password} scp ${exe_files} ${username}@${ip}:${curr_dir}
echo $?
echo "step 5: decompress files---------------------------------"
sshpass -p ${password} ssh ${username}@${ip} "cd ${curr_dir} && tar xzvf ${exe_files} 1>/dev/null && mv -f redis-server/* . && rm -rf redis-server 0</dev/null" 0</dev/null
echo $?
echo "step 6.${i}: update conf port ---------------------------------"
sshpass -p ${password} ssh ${username}@${ip} "cd ${curr_dir} && sed -i 's/^port[ 0-9]\+$/port ${port}/g' ${curr_dir}/conf/redis.conf 0</dev/null" 0</dev/null
echo $?
done < $1
exit 0
这个脚本的输入 ipportlist.txt文件格式为:
10.71.48.113:12000
10.71.48.114:12001
10.71.48.115:12002
10.71.48.114:12003
10.71.48.113:12004
10.71.48.115:12005
10.71.48.115:12006
10.71.48.113:12007
10.71.48.114:12008
6. 在启动集群之前,我们需要关闭待安装机器的防火墙。由于redis集群每个实例会开一个服务侦听端口和一个集群内通信端口,为了方便我们这里简单处理,直接写了个脚本把防火墙关闭了。也可以去iptables上配置好开放相应的端口。开关防火墙的脚本:
-
执行方法:
cat iplist.txt | sh firewall_control.sh start
#!/bin/bash
if [ $# -ne 1 ]
then
echo "Usage: $0 start|stop < iplistFile"
exit 1
fi
username='root'
password='123456'
while read ip
do
echo "at ${ip}"
if [ "${1}X" == "startX" ]
then
sshpass -p ${password} ssh ${username}@${ip} "service iptables start 0</dev/null" 0</dev/null
elif [ "${1}X" == "stopX" ]
then
sshpass -p ${password} ssh ${username}@${ip} "service iptables stop 0</dev/null" 0</dev/null
else
echo "invalid command!"
fi
done
echo "done."
exit 0
iplist.txt文件格式为:
10.71.48.113
10.71.48.114
10.71.48.115
7. 开启redis程序: 遍历所有实例的目录,cd到sbin下使用./redis-server -c ../conf/redis.conf启动redis实例, 批处理脚本redis_control.sh如下,包括了redis的开启/关闭/重启/存活检测。
-
启动: sh redis_control.sh ipportlist.txt start
-
关闭: sh redis_control.sh ipportlist.txt stop
#!/bin/bash
if [ $# -ne 2 ]
then
echo "Usage: $0 iplistFile start|stop|restart|check"
exit 1
fi
#启动所有redis实例
username='root'
password='123456'
dist_dir=/home/redis/redis-server
exe_dir="./redis-server"
echo ${dist_dir}
command=$2
while read ipport
do
ip=`echo ${ipport} |cut -d: -f1`
port=`echo ${ipport} | cut -d: -f2`
curr_dir="`echo ${dist_dir}|sed 's:/$::g'`-${port}"
bak_dir=`echo "${curr_dir}" | sed 's:/$::g'`_bak
echo "process host:${ip}:${port} at ${curr_dir}:${command}============================================="
if [ "${command}X" == "startX" ]
then
sshpass -p ${password} ssh ${username}@${ip} "cd ${curr_dir}/sbin && ./redis-server ../conf/redis.conf" 0</dev/null
elif [ "${command}X" == "stopX" ]
then
sshpass -p ${password} ssh ${username}@${ip} "cd ${curr_dir} && cat ./logs/redis.pid |xargs kill -9" 0</dev/null
elif [ "${command}X" == "restartX" ]
then
sshpass -p ${password} ssh ${username}@${ip} "cd ${curr_dir} && cat ./logs/redis.pid |xargs kill -9" 0</dev/null
sshpass -p ${password} ssh ${username}@${ip} "cd ${curr_dir}/sbin && ./redis-server ../conf/redis.conf" 0</dev/null
elif [ "${command}X" == "checkX" ]
then
sshpass -p ${password} ssh ${username}@${ip} "ps -ef |grep redis|grep ${port}|grep -v grep" 0</dev/null
else
echo "invalid command!"
fi
echo $?
done < $1
exit 0
8. 执行上面的start命令之后,只是启动了所有的实例,这些实例还没有互相连接起来组成集群。下面我们来看看如果把这些机器连接成集群。实际上redis官方提供了ruby脚本: ./redis-3.0.0/src/redis-trib.rb,来方便我们创建redis集群,不过需要安装相关的工具:
9. 安装好上述相关工具后就可以直接使用脚本redis-trib.rb安装了。同样,这里我们也对redis-trib.rb做了简单的封装,方便对参数的控制。封装之后的安装脚本为: redis_cluster_create.sh
-
执行: sh redis_cluster_create.sh ipportlist.txt
-
安装中根据提示输入yes即可
-
注意: –replicas 2表示每个主节点两个从节点
#!/bin/bash
if [ $# -ne 1 ]
then
echo "Usage: $0 ipportlist.txt"
exit 1
fi
#创建集群
all_addr=""
rb="./redis-3.0.0/src/redis-trib.rb"
all_addr=`xargs -n9 < $1`
echo ${all_addr}
echo "create cluster..."
${rb} create --replicas 2 ${all_addr}
echo "done."
exit 0
执行结果:
[root@xx106 redis-install]# sh redis_cluster_create.sh ipportlist.txt
10.71.48.113:12000 10.71.48.114:12001 10.71.48.115:12002 10.71.48.114:12003 10.71.48.113:12004 10.71.48.115:12005 10.71.48.115:12006 10.71.48.113:12007 10.71.48.114:12008
create cluster...
>>> Creating cluster
Connecting to node 10.71.48.113:12000: OK
Connecting to node 10.71.48.114:12001: OK
Connecting to node 10.71.48.115:12002: OK
Connecting to node 10.71.48.114:12003: OK
Connecting to node 10.71.48.113:12004: OK
Connecting to node 10.71.48.115:12005: OK
Connecting to node 10.71.48.115:12006: OK
Connecting to node 10.71.48.113:12007: OK
Connecting to node 10.71.48.114:12008: OK
>>> Performing hash slots allocation on 9 nodes...
Using 3 masters:
10.71.48.115:12002
10.71.48.114:12001
10.71.48.113:12000
Adding replica 10.71.48.114:12003 to 10.71.48.115:12002
Adding replica 10.71.48.113:12004 to 10.71.48.115:12002
Adding replica 10.71.48.115:12005 to 10.71.48.114:12001
Adding replica 10.71.48.115:12006 to 10.71.48.114:12001
Adding replica 10.71.48.114:12008 to 10.71.48.113:12000
Adding replica 10.71.48.113:12007 to 10.71.48.113:12000
M: 3bf182ba239438d96ef6ea4f3fa1a9b333d273d7 10.71.48.113:12000
slots:10923-16383 (5461 slots) master
M: 29569c90e2d11722f676bc8c650167ec6e6fcbe1 10.71.48.114:12001
slots:5461-10922 (5462 slots) master
M: bb9b7580b7b37b11ed1218af6e3c13794756a3f3 10.71.48.115:12002
slots:0-5460 (5461 slots) master
S: ae8baf40cf3741f9d4b6f3e880ba984867f1e66b 10.71.48.114:12003
replicates bb9b7580b7b37b11ed1218af6e3c13794756a3f3
S: 32962c506aa2e1a61c15b644a1750f0685739d7b 10.71.48.113:12004
replicates bb9b7580b7b37b11ed1218af6e3c13794756a3f3
S: 7ce9a69167e3c882db81539c6f01eb3de43b1526 10.71.48.115:12005
replicates 29569c90e2d11722f676bc8c650167ec6e6fcbe1
S: 3c949cb159bc2e74f49ccf1398ea6e3279d48656 10.71.48.115:12006
replicates 29569c90e2d11722f676bc8c650167ec6e6fcbe1
S: f6e15fad28fe15e4546cb984f226c7540388e698 10.71.48.113:12007
replicates 3bf182ba239438d96ef6ea4f3fa1a9b333d273d7
S: 6b3dc056e05e06ad5fb186881a2ad201a5ba8bb0 10.71.48.114:12008
replicates 3bf182ba239438d96ef6ea4f3fa1a9b333d273d7
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 10.71.48.113:12000)
M: 3bf182ba239438d96ef6ea4f3fa1a9b333d273d7 10.71.48.113:12000
slots:10923-16383 (5461 slots) master
M: 29569c90e2d11722f676bc8c650167ec6e6fcbe1 10.71.48.114:12001
slots:5461-10922 (5462 slots) master
M: bb9b7580b7b37b11ed1218af6e3c13794756a3f3 10.71.48.115:12002
slots:0-5460 (5461 slots) master
M: ae8baf40cf3741f9d4b6f3e880ba984867f1e66b 10.71.48.114:12003
slots: (0 slots) master
replicates bb9b7580b7b37b11ed1218af6e3c13794756a3f3
M: 32962c506aa2e1a61c15b644a1750f0685739d7b 10.71.48.113:12004
slots: (0 slots) master
replicates bb9b7580b7b37b11ed1218af6e3c13794756a3f3
M: 7ce9a69167e3c882db81539c6f01eb3de43b1526 10.71.48.115:12005
slots: (0 slots) master
replicates 29569c90e2d11722f676bc8c650167ec6e6fcbe1
M: 3c949cb159bc2e74f49ccf1398ea6e3279d48656 10.71.48.115:12006
slots: (0 slots) master
replicates 29569c90e2d11722f676bc8c650167ec6e6fcbe1
M: f6e15fad28fe15e4546cb984f226c7540388e698 10.71.48.113:12007
slots: (0 slots) master
replicates 3bf182ba239438d96ef6ea4f3fa1a9b333d273d7
M: 6b3dc056e05e06ad5fb186881a2ad201a5ba8bb0 10.71.48.114:12008
slots: (0 slots) master
replicates 3bf182ba239438d96ef6ea4f3fa1a9b333d273d7
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
done.
10. 运行:
[root@xx113 redis-server-12004]# ./sbin/redis-cli -c -p 12004
127.0.0.1:12004> set fuqingchuan abc
-> Redirected to slot [11708] located at 10.71.48.113:12000
OK
10.71.48.113:12000> set hello world
-> Redirected to slot [866] located at 10.71.48.115:12002
OK
10.71.48.115:12002> |