두대의 서버에 DRBD + GFS2 + Heartbeat + MariaDB를 설치하여 DB서버 이중화 구성을 하는 방법에 대해 다루겠습니다.
대충 서버 구성은 다음과 같다고 보시면 됩니다.
설치 순서는 DRBD -> GFS2 -> MariaDB -> Heartbeat 가 되겠습니다.
설명 중간 중간에 “[root@ruo91 ~]#” 의 형태로 나오면 양쪽 서버 모두 포함해서 설정하는 부분 입니다.. 실수 없으시길..
테스트 환경
CentOS 6.5 x86_64 (2.6.32-431.5.1.el6)
서버 구성
Host Name Type IP Storage
ruo91-node1 Primary 192.168.0.10 10G SSD
ruo91-node2 Secondary 192.168.0.11 10G SSD
1. Host Name 별칭 설정 (Both)
/etc/hosts 파일에 서버의 별칭을 설정 합니다.
[root@ruo91 ~]# nano /etc/hosts # DRBD 192.168.0.10 ruo91-node1 192.168.0.11 ruo91-node2
2. NTP 동기화 (Both)
서버의 날짜 및 시간이 맞지 않으면 동기화가 되지 않을수 있기 때문에 crontab을 이용하여 주기 적으로 동기화 될수 있도록 설정 합니다.
[root@ruo91 ~] # nano /etc/crontab # NTP 동기화 # 매 1분마다 동기화를 진행 하므로 NTP서버를 직접 구축해서 사용하는 것을 권장합니다. 1 * * * * root ntpdate 1.kr.pool.ntp.org
3. DRBD(Distributed Replicated Block Device)
– DRBD 설치 (Both)
elrepo의 저장소를 추가 하고 설치 합니다.
[root@ruo91 ~] # rpm -ivh http://elrepo.org/elrepo-release-6-5.el6.elrepo.noarch.rpm [root@ruo91 ~] # yum install -y kmod-drbd84 drbd84-utils
수동으로 DRBD Module을 적재 시킵니다.
[root@ruo91 ~] # modprobe drbd minor_count=1
부팅시 자동으로 모듈을 적재 시키도록 설정 파일을 하나 작성 합니다.
[root@ruo91 ~]# echo "modprobe minor minor_count=1" >> /etc/rc.local
– DRBD 설정 (Both)
“drbd_disk1.res”라는 이름으로 설정 파일을 하나 생성하고 아래와 같이 사용자 환경에 맞게 변경합니다.
[root@ruo91 ~] # nano /etc/drbd.d/drbd_disk1.res
become-primary-on both;
resource drbd_disk1 {
protocol C;
startup {
wfc-timeout 60;
outdated-wfc-timeout 40;
degr-wfc-timeout 60;
}
net {
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
cram-hmac-alg sha1;
shared-secret sync_disk;
}
disk {
fencing resource-and-stonith;
}
handlers {
fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
}
syncer {
rate 100M;
verify-alg sha1;
}
on ruo91-node1 {
device /dev/drbd0;
disk /dev/sdb;
address 192.168.0.10:7789;
meta-disk internal;
}
on ruo91-node2 {
device /dev/drbd0;
disk /dev/sdb;
address 192.168.0.11:7789;
meta-disk internal;
}
}
– DRBD Meta data 생성 (Both)
[root@ruo91 ~]# drbdadm create-md drbd_disk1 riting meta data... initializing activity log NOT initializing bitmap New drbd meta data block successfully created. success
– DRBD 실행 (Both)
[root@ruo91 ~]# /etc/init.d/drbd start Starting DRBD resources: [ create res: drbd_disk1 prepare disk: drbd_disk1 adjust disk: drbd_disk1 adjust net: drbd_disk1 ] ..........
– 동기화 설정
– ruo91-node1
ruo91-node1 서버를 primary로 지정 하겠습니다.
[root@ruo91-node1 ~]# drbdadm primary --force drbd_disk1
이후 primary <-> Secondary 동기화가 완료 될때까지 기다립니다.
[root@ruo91-node1 ~]# drbd-overview 0:drbd_disk1/0 SyncSource Primary/Secondary UpToDate/Inconsistent C r----- [=================>..] sync'ed: 92.2% (800/10236)M
동기화가 완료 되면 이런식으로 나와야 합니다.
[root@ruo91-node1 ~]# drbd-overview 0:drbd_disk1/0 Connected Primary/Secondary UpToDate/UpToDate C r-----
– ruo91-node2
Primary 로 변경 합니다.
[root@ruo91-node2 ~]# drbdadm primary drbd_disk1 [root@ruo91-node2 ~]# drbd-overview 0:drbd_disk1/0 Connected Primary/Primary UpToDate/UpToDate C r-----
4. Cluster + GFS2(Global File System)
– Cluster Software 설치 및 설정 (Both)
[root@ruo91 ~]# rpm -ivh http://mirror.premi.st/epel/6/i386/epel-release-6-8.noarch.rpm [root@ruo91 ~]# yum update [root@ruo91 ~]# yum groupinstall -y "High Availability" "Resilient Storage" "High Availability Management"
cluster.conf 설정 파일을 하나 만들어 줍니다.
[root@ruo91 ~]# nano /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="1" name="drbd-cluster">
<rm log_level="4"/>
<fence_daemon clean_start="1" post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="ruo91-node1" nodeid="1" votes="1">
<fence>
<method name="2">
<device name="LastResortNode01"/>
</method>
</fence>
</clusternode>
<clusternode name="ruo91-node2" nodeid="2" votes="1">
<fence>
<method name="2">
<device name="LastResortNode02"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1"/>
<fencedevices>
<fencedevice agent="fence_manual" name="LastResortNode01" nodename="ruo91-node1"/>
<fencedevice agent="fence_manual" name="LastResortNode02" nodename="ruo91-node2"/>
</fencedevices>
<totem consensus="4800" join="60" token="10000" token_retransmits_before_loss_const="20"/>
</cluster>
cluster management 를 실행합니다.
[root@ruo91 ~]# service cman start Starting cluster: Checking if cluster has been disabled at boot... [ OK ] Checking Network Manager... [ OK ] Global setup... [ OK ] Loading kernel modules... [ OK ] Mounting configfs... [ OK ] Starting cman... [ OK ] Waiting for quorum... [ OK ] Starting fenced... [ OK ] Starting dlm_controld... [ OK ] Tuning DLM kernel config... [ OK ] Starting gfs_controld... [ OK ] Unfencing self... [ OK ] Joining fence domain... [ OK ]
부팅시 자동 실행 되도록 설정합니다.
[root@ruo91 ~]# chkconfig --level 2345 cman on
– GFS2 파일 시스템 포멧 (Only one of the nodes)
이 부분은 ruo91-node1에서만 진행 하겠습니다.
[root@ruo91-node1 ~]# mkfs.gfs2 -p lock_dlm -t drbd-cluster:drbd_disk1 /dev/drbd0 -j 2 This will destroy any data on /dev/drbd0. It appears to contain: Linux rev 1.0 ext4 filesystem data (extents) (large files) (huge files) Are you sure you want to proceed? [y/n] y Device: /dev/drbd0 Blocksize: 4096 Device Size 10.00 GB (2621351 blocks) Filesystem Size: 10.00 GB (2621351 blocks) Journals: 2 Resource Groups: 40 Locking Protocol: "lock_dlm" Lock Table: "drbd-cluster:drbd_disk1" UUID: d2597223-8493-25da-9293-89be31c8f3f8
– GFS2 마운트
[root@ruo91-node1 ~]# mkdir /storage_gfs2 [root@ruo91-node1 ~]# mount.gfs2 -o noatime,nodiratime /dev/drbd0 /storage_gfs2
확인
[root@ruo91-node1 ~]# drbd-overview 0:drbd_disk1/0 Connected Primary/Primary UpToDate/UpToDate C r----- /storage_gfs2 gfs2 10G 259M 9.8G 3%
[root@ruo91-node1 ~]# cat /var/log/messages Mar 4 22:43:32 ruo91-node1 kernel: Slow work thread pool: Starting up Mar 4 22:43:32 ruo91-node1 kernel: Slow work thread pool: Ready Mar 4 22:43:32 ruo91-node1 kernel: GFS2 (built Feb 11 2014 21:58:54) installed Mar 4 22:43:32 ruo91-node1 kernel: GFS2: fsid=: Trying to join cluster "lock_dlm", "drbd-cluster:drbd_disk1" Mar 4 22:43:32 ruo91-node1 kernel: dlm: Using TCP for communications Mar 4 22:43:32 ruo91-node1 kernel: GFS2: fsid=drbd-cluster:drbd_disk1.0: Joined cluster. Now mounting FS... Mar 4 22:43:32 ruo91-node1 kernel: GFS2: fsid=drbd-cluster:drbd_disk1.0: jid=0, already locked for use Mar 4 22:43:32 ruo91-node1 kernel: GFS2: fsid=drbd-cluster:drbd_disk1.0: jid=0: Looking at journal... Mar 4 22:43:32 ruo91-node1 kernel: GFS2: fsid=drbd-cluster:drbd_disk1.0: jid=0: Done Mar 4 22:43:32 ruo91-node1 kernel: GFS2: fsid=drbd-cluster:drbd_disk1.0: jid=1: Trying to acquire journal lock... Mar 4 22:43:32 ruo91-node1 kernel: GFS2: fsid=drbd-cluster:drbd_disk1.0: jid=1: Looking at journal... Mar 4 22:43:33 ruo91-node1 kernel: GFS2: fsid=drbd-cluster:drbd_disk1.0: jid=1: Done
5. MariaDB 설치 및 설정 (Both)
– MariaDB 저장소 추가
[root@ruo91 ~]# nano /etc/yum.repos.d/mariadb.repo # MariaDB 5.5 CentOS repository list - created 2014-03-04 12:11 UTC # http://mariadb.org/mariadb/repositories/ [mariadb] name = MariaDB baseurl = http://yum.mariadb.org/5.5/centos6-amd64 gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB gpgcheck=1
MariaDB 설치
[root@ruo91 ~]# yum update && yum install -y MariaDB-server MariaDB-client MariaDB-devel MariaDB-shared
기존 설정파일 삭제
[root@ruo91 ~]# rm -f /etc/my.cnf.d/*
my-huge.cnf 설정 파일 사용
[root@ruo91 ~]# ln -s /usr/share/mysql/my-huge.cnf /etc/my.cnf.d/mariadb.cnf
/etc/init.d/mysql의 datadir 변경
[root@ruo91 ~]# nano /etc/init.d/mysql # If you change base dir, you must also change datadir. These may get # overwritten by settings in the MySQL configuration files. basedir= datadir=/storage_gfs2/mariadb-data
Database 설치
[root@ruo91 ~]# mysql_install_db --datadir=/storage_gfs2/mariadb-data [root@ruo91 ~]# chown -R mysql:mysql /storage_gfs2/mariadb-data
MariaDB 시작
[root@ruo91 ~]# service mysql start Starting MySQL......... SUCCESS!
root 비밀번호 설정
[root@ruo91 ~]# mysqladmin -u root password 'your-passwords'
사용자 데이터베이스 추가
[root@ruo91 ~]# mysql -u root -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 3 Server version: 5.5.36-MariaDB-log MariaDB Server Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> MariaDB [(none)]> CREATE DATABASE ruo91 CHARACTER SET utf8;
웹서버로 사용하고 있는 IP 에서 접근이 가능 하도록 설정
MariaDB [(none)]> CREATE USER 'ruo91'@'%' IDENTIFIED BY 'your-passwords'; MariaDB [(none)]> GRANT ALL PRIVILEGES ON ruo91.* TO 'ruo91'@'%';
6. Heartbeat
– Heartbeat 설치 (Both)
[root@ruo91 ~]# yum install -y heartbeat*
ha.cf 설정 파일을 하나 만듭니다.
[root@ruo91 ~]# nano /etc/ha.d/ha.cf # 로그 use_logd yes logfile /var/log/ha.log debugfile /var/log/ha-debug.log # 유지 시간 keepalive 2 # 호스트가 죽었다고 판단하는 시간 deadtime 10 # 자동 장애 복구 auto_failback on initdead 10 bcast eth0 # Master node ruo91-node1 # Slave node ruo91-node2
sha1를 사용 하도록 설정 합니다.
[root@ruo91 ~]# cat /etc/ha.d/authkeys # 인증에 관한 설정 auth 2 2 sha1 yongbok-cluster [root@ruo91 ~]# chmod 600 /etc/ha.d/authkeys
각각의 노드에 haresources 파일을 생성합니다.
[root@ruo91-node1 ~]# nano /etc/ha.d/haresources ruo91-node1 drbddisk::drbd_disk1 \ Filesystem::/dev/drbd0::/storage_gfs2::gfs2::noatime,nodiratime \ mysql \ IPaddr2::192.168.0.12/24/eth0/192.168.0.255
[root@ruo91-node2 ~]# nano /etc/ha.d/haresources ruo91-node2 drbddisk::drbd_disk1 \ Filesystem::/dev/drbd0::/storage_gfs2::gfs2::noatime,nodiratime \ mysql \ IPaddr2::192.168.0.12/24/eth0/192.168.0.255
Heartbeat에 MariaDB 서비스를 등록
[root@ruo91 ~]# ln -s /etc/init.d/mysql /etc/ha.d/resource.d/mysql
MariaDB, drbd를 중지 합니다.
[root@ruo91 ~]# service mysql stop [root@ruo91 ~]# service drbd stop
부팅시 Heartbeat가 해당 서비스를 컨트롤 하므로 runlevel을 모두 해제 합니다.
[root@ruo91 ~]# chkconfig --level 2345 drbd off [root@ruo91 ~]# chkconfig --level 2345 mysql off
Heartbeat를 시작합니다.
[root@ruo91 ~]# service heartbeat start Starting High-Availability services: Done.
이후 eth0 인터페이스에 가상의 IP가 하나더 추가 되어있는 것을 볼수 있습니다.
[root@ruo91 ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:1f:e1:3c brd ff:ff:ff:ff:ff:ff
inet 192.168.0.10/24 brd 192.168.0.255 scope global eth0
inet 192.168.0.12/24 brd 192.168.0.255 scope global secondary eth0
inet6 fe80::a00:27ff:fe1f:e13c/64 scope link
valid_lft forever preferred_lft forever
위와 같이 잘 설정 되었다면 서버 장애가 발생해도 auto-failover를 잘 수행하게 될것 같습니다.
Thanks -:)