【MySQL】MHA部署與MasterFailover代碼分析

官網：https://code.google.com/p/mysql-master-ha/

參考：http://blog.csdn.net/wulantian/article/details/13287975

參考：http://www.cnblogs.com/wingsless/p/4033093.html

參考：http://www.cnblogs.com/xuanzhi201111/p/4231412.html#jtss-tsina

參考：http://ylw6006.blog.51cto.com/470441/890360/

參考：http://os.51cto.com/art/201307/401702.htm

參考：http://www.it165.net/database/html/201508/13780.html

下載：http://pan.baidu.com/s/1pJ0VkSz

部署：

[root@dns packages]# tar -zxvf mha4mysql-manager-0.53.tar.gz [root@dns mha4mysql-manager-0.53]# perl Makefile.PL [root@dns mha4mysql-manager-0.53]# make && make install 配置有很多參數，可以看看lib\MHA下面的Config.pm代碼 [root@dns mha4mysql-manager-0.53]# vim /etc/3307.cnf [server default] manager_log=/data1/masterha/3307_leju/manager.log manager_workdir=/data1/masterha/3307_leju multi_tier_slave=3 password=passwd ping_interval=1 repl_password=x8uf7nbv5x64 repl_user=repl shutdown_script="/samples/scripts/power_manager" //設置故障發生后關閉故障主機腳本（該腳本的主要作用是關閉主機放在發生腦裂
master_pid_file=/data1/mysql/3307_leju/mysql.pid //shutdown_script腳本中需要使用的pid文件位置，通過pid文件中的pid進程號強制殺掉主庫實例 ssh_user=root user=ha[server1] hostname=10.207.0.125 master_binlog_dir="/data1/mysql/3307_leju" port=3307[server2] hostname=10.207.0.126 master_binlog_dir="/data1/mysql/3307_leju" port=3307[server3] hostname=10.207.0.127 master_binlog_dir="/data1/mysql/3307_leju" port=3307[server4] hostname=10.207.0.128 master_binlog_dir="/data1/mysql/3307_leju" port=3307 啟動前檢查復制關系 [root@dns bin]# masterha_check_repl --conf=/etc/masterha/3307.cnf 啟動manager masterha_manager啟動有很多參數，參數說明不列舉說明
[root@dns bin]# masterha_manager --conf=/etc/masterha/3307.cnf --remove_dead_master_conf --ignore_last_failover

功能分析：

注意MHA需要super,select,replication slave,replication client權限，在測試MHA0.56時由于權限不夠perl也沒有相關報錯，導致failover卡住，問題很難才定位到

MHA的故障切換是由lib/MHA/MasterFailover.pm代碼完成。

編譯后位置移動到/usr/local/share/perl5/MHA/MasterFailover.pm。

代碼主體為do_master_failover函數完成，代碼片段：

sub do_master_failover {my $error_code = 1;my ( $dead_master, $new_master );eval {my @servers_config = init_config();$log->info("Starting master failover.");$log->info("* Phase 1: Configuration Check Phase..\n");
$dead_master = check_settings( \@servers_config );$log->info("** Phase 1: Configuration Check Phase completed.\n");$log->info("* Phase 2: Dead Master Shutdown Phase..\n");
force_shutdown($dead_master);$log->info("* Phase 2: Dead Master Shutdown Phase completed.\n");$log->info("* Phase 3: Master Recovery Phase..\n"); $log->info("* Phase 3.1: Getting Latest Slaves Phase..\n");
check_set_latest_slaves();$log->info("* Phase 3.2: Saving Dead Master's Binlog Phase..\n");
save_master_binlog($dead_master);$log->info("* Phase 3.3: Determining New Master Phase..\n");
my $latest_base_slave = find_latest_base_slave($dead_master);$new_master = select_new_master( $dead_master, $latest_base_slave );my ( $master_log_file, $master_log_pos ) = recover_master( $dead_master, $new_master, $latest_base_slave );$new_master->{activated} = 1;$log->info("* Phase 3: Master Recovery Phase completed.\n");$log->info("* Phase 4: Slaves Recovery Phase..\n");$error_code = recover_slaves( $dead_master, $new_master, $latest_base_slave, $master_log_file, $master_log_pos);if ( $g_remove_dead_master_conf && $error_code == 0 ) { MHA::Config::delete_block_and_save( $g_config_file, $dead_master->{id}, $log );}cleanup();}; ... }

可以通過增加幾行代碼實現MHA切換通知DNS達到前端請求的切換。

# 建立連接 my $dbh = DBI->connect("DBI:mysql:database=dblayer;mysql_socket=/tmp/mysql_3307.sock","dblayer","dblayer"); # 下線主庫域名，IsLive狀態修改（Bind_DLZ的MySQL驅動查詢增加IsLive字段判斷） my $rows = $dbh->do("UPDATE dns_records SET IsLive=0 WHERE data='$dead_master->{ip}' and port='$dead_master->{port}'"); # 獲取主庫host（Bind的dns_records表中使用IP和PORT對應一個唯一HOST，多實例的支持） my $sth = $dbh->prepare("select host from dns_records where data='$dead_master->{ip}' and port='$dead_master->{port}'"); $sth->execute(); my $ref = $sth->fetchrow(); # 將新主庫更新host my $rows = $dbh->do("UPDATE dns_records SET HOST='$ref' WHERE data='$new_master->{ip}' and port='$dead_master->{port}'"); # 將切換記錄到庫 my $rows = $dbh->do("INSERT INTO masterfailover_log(old_master_ip,new_master_ip,port,type) values ('$dead_master->{ip}','$new_master->{ip}','$dead_master->{port}',1)"); # 斷開連接
$sth->finish;
$dbh->disconnect();

在加入shutdown_script后，出現報錯：

Thu Sep 24 14:21:42 2015 - [info] /usr/local/mha_manager/samples/scripts/power_manager --command=status --ssh_user=root --host=10.207.0.125 --ip=10.207.0.125 Undefined subroutine &main::FIXME_xxx called at /usr/local/mha_manager/samples/scripts/power_manager line 387. Thu Sep 24 14:21:42 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln235] Failed to get power status with return code 1:0. Thu Sep 24 14:21:42 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln383] Error happend on checking configurations. at /usr/local/bin/masterha_check_repl line 48 Thu Sep 24 14:21:42 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln478] Error happened on monitoring servers. Thu Sep 24 14:21:42 2015 - [info] Got exit code 1 (Not master dead).MySQL Replication Health is NOT OK!

打開power_manager可以看到，報錯的部分是一個未定義函數FIXME_xxx，代碼是想通過服務器管理地址進行操作，嘗試將未定義函數部分去掉會出現更多問題。

在power_manager代碼下面有一些注釋說明，說明中有：

# killing mysqld with specified pid file. This is useful when you run multiple MySQL instances and want to stop only specified instance power_manager --command=stopssh --host=master_server --ssh_user=root --pid_file=/var/lib/mysql/mysqld.pid

所以不在配置中添加shutdown_script，直接在MasterFailover.pm中force_shutdown_internal函數最后添加power_manager的調用。

system("/usr/local/mha_manager/samples/scripts/power_manager --command=stopssh --host='$dead_master->{ip}' --ssh_user=root --pid_file='$dead_master->{master_pid_file}'");

因為power_manager有未定義的函數，如果直接調用，在異常主庫可以ssh通但是pid已經不存在或者根本ssh不通的情況下，因為stopssh函數返回值異常仍會報錯未命名函數，所以改一下power_manager，注釋掉一部分代碼，防止killall -9 mysql mysqld。

sub stopssh {...if ($pid_file) {$command = "\"if [ ! -e $pid_file ]; then exit 1; fi; pid=\\\`cat $pid_file\\\`; rm -f $pid_file; kill -9 \\\$pid; a=\\\`ps ax | grep $pid_file | grep -v grep | wc | awk {'print \\\$1'}\\\`; if [ \"a\\\$a\" = \"a0\" ]; then exit 10; fi; sleep 1; a=\\\`ps ax | grep $pid_file | grep -v grep | wc | awk {'print \\\$1'}\\\`; if [ \"a\\\$a\" = \"a0\" ]; then exit 10; else exit 1; fi\"";( $high_ret, $low_ret ) = MHA::ManagerUtil::exec_system("ssh $ssh_user_host $MHA::ManagerConst::SSH_OPT_CHECK $command");if ( $high_ret == $SSH_STOP_OK && $low_ret == 0 ) {print "ssh reachable. mysqld stopped. power off not needed.\n";return $high_ret;}elsif ( $high_ret == 1 && $low_ret == 0 ) {print "ssh reachable. not found $pid_file,mysqld stopped. power off not needed.\n";return 10;}else {print "Killing mysqld instance based on $pid_file failed.\n"; return 10;} }# 后面killall -9 mysql mysqld 注釋 }

在配置里面加report_script=/usr/local/bin/send_report后

修改mha_manager/samples/scripts/send_report的代碼可以通知ZABBIX和郵件通知。ZABBIX在mha_manager主機加上一個監控項，把Item設置為Zabbix trapper，Type of information設置為字符

my @array=split(/ /,$subject); my $sender=join('',$array[0],$array[7]);system("/data1/scripts/zabbix_sender --zabbix-server serverip --port 10051 --host 'mhaserver' --key mysql.MHA --value '$sender'"); system("/bin/echo '$body' | /usr/bin/mutt -s '$subject' 'jiangxu\@test.com'");

?

posted on 2015-05-08 11:42 poorX 閱讀(...) 評論(...) 編輯 收藏

轉載于:https://www.cnblogs.com/jiangxu67/p/4487248.html

總結

以上是生活随笔為你收集整理的【MySQL】MHA部署与MasterFailover代码分析的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。