當前位置：首頁 > 运维知识 > 数据库 >内容正文

数据库

mysql中shift h_MySQL复制技术对比与容器化探究

發布時間：2025/3/19 数据库 21 豆豆

生活随笔收集整理的這篇文章主要介紹了 mysql中shift h_MySQL复制技术对比与容器化探究小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

MySQL和MariaDB

我們都知道，MariaDB是MySQL的一個分支，那么他們之間的區別在哪呢？

MariaDB和Mysql的主要區別：MariaDB有12個新的存儲引擎，而MySQL有較少的存儲引擎。

MariaDB具有更大的連接池，最多支持200,000個以上的連接，而MySQL具有較小的連接池。

在MariaDB中，復制速度較快，而在MySQL中，復制速度較慢。

MariaDB是開源的，而MySQL在其企業版中使用了一些專有代碼。

MariaDB不支持數據屏蔽和動態列，而MySQL支持。

相對而言，MariaDB比MySQL快。

從市場份額看，MySQL比MariaDB還是要高非常多。

MySQL和MariaDB的復制技術

MySQL最簡單的是異步復制，如下圖所示：主實例可讀寫，備實例可讀。

主備復制的架構圖如下：傳統的數據主從輔助屬于異步復制，從庫起IO線程連接主庫，獲取主庫二進制日志寫到本地中繼日志，并更新master-info文件(存放主庫相關信息)，從庫再利用SQL線程執行中繼日志。

半同步復制是建立在基本的主從復制基礎上，利用插件完成半同步復制，傳統的主從復制，不管從庫是否正確獲取到二進制日志，主庫不斷更新，半同步復制則當確認了從庫把二進制日志寫入中繼日志才會允許提交，如果從庫遲遲不返回ack,主庫會自動將半同步復制狀態取消，進入最基本的主從復制模式。

第二種技術是組復制，簡稱MGR：

MGR是MySQL官方于2016年12月推出的一個全新的高可用與高擴展的解決方案。MGR是MySQL官方在5.7.17版本引進的一個數據庫高可用與高擴展的解決方案，以插件形式提供，實現了分布式下數據的最終一致性, 它是MySQL5.7版本出現的新特性，它提供了高可用、高擴展、高可靠的MySQL集群服務。

組復制是一種可用于實現容錯系統的技術。復制組是一個通過消息傳遞相互交互的server集群。復制組由多個server成員組成，如上圖的master1，master2，master3，所有成員獨立完成各自的事務。當客戶端先發起一個更新事務，該事務先在本地執行，執行完成之后就要發起對事務的提交操作了。在還沒有真正提交之前需要將產生的復制寫集廣播出去，復制到其他成員。如果沖突檢測成功，組內決定該事務可以提交，其他成員可以應用，否則就回滾。最終，這意味著所有組內成員以相同的順序接收同一組事務。因此組內成員以相同的順序應用相同的修改，保證組內數據強一致性。

新成員加入組的簡單流程：

當有新的成員加入組中，組內原有的成員會在二進制日志中插入一個視圖切換的事件。

在組成員內找到一個donor捐贈之前缺失的數據，如果這個donor突然下線了，新成員會從新的donor獲取缺失的數據，這時候組還在不斷更新，新成員會將新的事件寫到內存的一個臨時空間

當獲取到視圖切換事件的時候，新成員將開始執行保存到內存臨時空間的事件

MariaDB的復制主要有主備復制(和MySQL類似)和Galera兩種，目前沒看到類似MGR的技術。這里不再展開討論。

OpenShift提供的MySQL容器鏡像

如果我們想要實現MySQL的容器化，怎么做的？

紅帽OpenShift提供MySQL的容器化鏡像，可以看到最新的是MySQL 8.0.

容器鏡像安全程度很高：

查看鏡像dockerfile的部分內容，鏡像基于RHEL8.1

查看容器鏡像的package list：

以最簡單的方式運行MySQL容器鏡像

使用紅帽提供的MySQL鏡像，最簡單的方法是直接運行(實驗的時候，我們使用基于rhel7的mysql8鏡像)：podman?run?-d?--name?mysql_database?-e?MYSQL_USER=user?-e?MYSQL_PASSWORD=pass?-e?MYSQL_DATABASE=db?-p?3306:3306?rhscl/mysql-80-rhel7

查看運行的mysql容器：

上面展示的是最簡單的運行mysql容器鏡像的方式。這個鏡像包含的環境變量和解釋如下。

MYSQL_USERUser name for MySQL account to be createdMYSQL_PASSWORDPassword for the user accountMYSQL_DATABASEDatabase nameMYSQL_ROOT_PASSWORDPassword for the root user (optional)The following environment variables influence the MySQL configuration file. They are all optional.MYSQL_LOWER_CASE_TABLE_NAMES (default: 0)Sets how the table names are stored and comparedMYSQL_MAX_CONNECTIONS (default: 151)The maximum permitted number of simultaneous client connectionsMYSQL_MAX_ALLOWED_PACKET (default: 200M)The maximum size of one packet or any generated/intermediate stringMYSQL_FT_MIN_WORD_LEN (default: 4)The minimum length of the word to be included in a FULLTEXT indexMYSQL_FT_MAX_WORD_LEN (default: 20)The maximum length of the word to be included in a FULLTEXT indexMYSQL_AIO (default: 1)Controls the innodb_use_native_aio setting value in case the native AIO is broken. See http://help.directadmin.com/item.php?id=529MYSQL_TABLE_OPEN_CACHE (default: 400)The number of open tables for all threadsMYSQL_KEY_BUFFER_SIZE (default: 32M or 10% of available memory)The size of the buffer used for index blocksMYSQL_SORT_BUFFER_SIZE (default: 256K)The size of the buffer used for sortingMYSQL_READ_BUFFER_SIZE (default: 8M or 5% of available memory)The size of the buffer used for a sequential scanMYSQL_INNODB_BUFFER_POOL_SIZE (default: 32M or 50% of available memory)The size of the buffer pool where InnoDB caches table and index dataMYSQL_INNODB_LOG_FILE_SIZE (default: 8M or 15% of available memory)The size of each log file in a log groupMYSQL_INNODB_LOG_BUFFER_SIZE (default: 8M or 15% of available memory)The size of the buffer that InnoDB uses to write to the log files on diskMYSQL_DEFAULTS_FILE (default: /etc/my.cnf)Point to an alternative configuration fileMYSQL_BINLOG_FORMAT (default: statement)Set sets the binlog format, supported values are row and statementMYSQL_LOG_QUERIES_ENABLED (default: 0)To enable query logging set this to 1MYSQL_DEFAULT_AUTHENTICATION_PLUGIN (default: caching_sha2_password)Set default authentication plugin. Accepts values mysql_native_password or caching_sha2_password.You can also set the following mount points by passing the -v /host:/container flag to Docker./var/lib/mysql/dataMySQL data directory當使用--memory參數集運行MySQL映像并且您未為某些參數指定值時，它們的值將根據可用內存自動計算。MYSQL_KEY_BUFFER_SIZE (default: 10%)key_buffer_sizeMYSQL_READ_BUFFER_SIZE (default: 5%)read_buffer_sizeMYSQL_INNODB_BUFFER_POOL_SIZE (default: 50%)innodb_buffer_pool_sizeMYSQL_INNODB_LOG_FILE_SIZE (default: 15%)innodb_log_file_sizeMYSQL_INNODB_LOG_BUFFER_SIZE (default: 15%)innodb_log_buffer_size

模板部署mysql

在上面小節中，我們使用pod直接運行mysql容器鏡像，在生產上這樣做顯然是不合適的。我們可以通過mysql的模板在OCP上進行部署：

可以直接部署mysql 8.0：

部署成功：

# oc get pods |grep -i davidmysql-david-1-deploy ? ? ?0/1 ? ? Completed ? 0 ? ? ? ? ?57smysql-david-1-qfvqm ? ? ? 1/1 ? ? Running ? ? 0 ? ? ? ? ?49s?oc?new-app?mysql-80-rhel7:30~https://github.com/sclorg/mysql-container.git?\

這種部署的方式也很方便。

定制化部署mysql

接下來，我們展示定制化部署mysql。使用S2I的方式進行定制化。builder image就是本文開頭展示的mysql8.0鏡像。s2i的源碼地址：https://github.com/sclorg/mysql-container.git

首先導入紅帽mysql容器鏡像的is:

oc?import-image?rhscl/mysql-80-rhel7?--from=registry.access.redhat.com/rhscl/mysql-80-rhel7?--confirm?--all

接下來，利用image stream定制化部署mysql：

oc?new-app?mysql-80-rhel7:30~https://github.com/sclorg/mysql-container.git?\--name?my-mysql-rhel7?\--context-dir=examples/extend-image?\--env?MYSQL_OPERATIONS_USER=opuser?\--env?MYSQL_OPERATIONS_PASSWORD=oppass?\--env?MYSQL_DATABASE=opdb?\--env?MYSQL_USER=user?\--env?MYSQL_PASSWORD=pass

我們查看上面輸入參數examples/extend-image對應的目錄：

mysql-cfg/?啟動容器時，該目錄中的文件將用作mysqld守護程序的配置。在此文件上運行envsubst命令以仍然允許使用環境變量自定義鏡像:

mysql-container/examples/extend-image/mysql-cfg/myconfig.cnf[mysqld]stored_program_cache?=?524288--env?MYSQL_USER=user?\--env?MYSQL_PASSWORD=pass

mysql-pre-init/?此目錄中可用的Shell腳本(* .sh)是在啟動mysqld守護程序之前獲取的。

mysql-container/examples/extend-image/mysql-data/init.sqlCREATE TABLE products (id INTEGER, name VARCHAR(256), price FLOAT, variant INTEGER);CREATE TABLE products_variant (id INTEGER, name VARCHAR(256));INSERT INTO products_variant (id, name) VALUES ('1', 'blue'), ('2', 'green');

mysql-init/?在本地啟動mysqld守護程序時，將獲得此目錄中可用的mysql-init / Shell腳本(* .sh)。在此階段，使用$ {mysql_flags}連接到本地運行的守護程序，例如mysql $ mysql_flags

mysql-container/examples/extend-image/mysql-init/80-add-arbitrary-users.shcreate_arbitrary_users() {# Do not care what option is compulsory here, just create what is specifiedlog_info "Creating user specified by MYSQL_OPERATIONS_USER (${MYSQL_OPERATIONS_USER}) ..."mysql $mysql_flags <

可以在提供給s2i的腳本中使用的變量：

mysql工具的$ mysql_flags參數，它將在初始化期間連接到本地運行的mysqld

$ MYSQL_RUNNING_AS_MASTER變量在使用run-mysqld-master命令運行容器時定義

$ MYSQL_RUNNING_AS_SLAVE變量，在使用run-mysqld-slave命令運行容器時定義

從空數據目錄初始化容器時定義的$ MYSQL_DATADIR_FIRST_INIT變量

在s2i構建期間，將所有提供的文件復制到/opt/app-root/src目錄中，并復制到結果鏡像中。如果目標目錄中存在某些配置文件，則具有相同名稱的文件將被覆蓋。同樣，只有一個具有相同名稱的文件可用于自定義，并且用戶提供的文件比/usr/share/container-scripts/mysql /-中的默認文件更可取，因此可以覆蓋它們。

查看mysql部署部分日志：

[root@lb.davidwei-cluster1 /opt/davidwei-cluster1/data]# oc logs -f my-mysql-rhel7-2-5lnts=> sourcing 20-validate-variables.sh ...=> sourcing 25-validate-replication-variables.sh ...=> sourcing 30-base-config.sh ...---> 16:35:48 ? ? Processing basic MySQL configuration files ...=> sourcing 60-replication-config.sh ...=> sourcing 70-s2i-config.sh ...---> 16:35:48 ? ? Processing additional arbitrary ?MySQL configuration provided by s2i ...=> sourcing 20-default-authentication-plugin.cnf ...=> sourcing 40-paas.cnf ...=> sourcing 50-my-tuning.cnf ...---> 16:35:48 ? ? Initializing database ...---> 16:35:48 ? ? Running /opt/rh/rh-mysql80/root/usr/libexec/mysqld --initialize --datadir=/var/lib/mysql/data---> 16:35:53 ? ? Starting MySQL server with disabled networking ...---> 16:35:53 ? ? Waiting for MySQL to start ...---> 16:35:54 ? ? Waiting for MySQL to start ...2020-03-27T16:35:53.690350Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.2020-03-27T16:35:53.700015Z 0 [System] [MY-010116] [Server] /opt/rh/rh-mysql80/root/usr/libexec/mysqld (mysqld 8.0.17) starting as process 852020-03-27T16:35:54.410486Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.2020-03-27T16:35:54.459936Z 0 [System] [MY-010931] [Server] /opt/rh/rh-mysql80/root/usr/libexec/mysqld: ready for connections. Version: '8.0.17' ?socket: '/tmp/mysql.sock' ?port: 0 ?Source distribution.2020-03-27T16:35:54.486976Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/lib/mysql/mysqlx.sock'---> 16:35:55 ? ? MySQL started successfullymysql: [Warning] Using a password on the command line interface can be insecure.The mysql_upgrade client is now deprecated. The actions executed by the upgrade client are now done by the server.To upgrade, please start the new MySQL binary with the older data directory. Repairing user tables is done automatically. Restart is not required after upgrade.The upgrade process automatically starts on running a new MySQL binary with an older data directory. To avoid accidental upgrades, please use the --upgrade=NONE option with the MySQL binary. The option --upgrade=FORCE is also provided to run the server upgrade sequence on demand.It may be possible that the server upgrade fails due to a number of reasons. In that case, the upgrade sequence will run again during the next MySQL server start. If the server upgrade fails repeatedly, the server can be started with the --upgrade=MINIMAL option to start the server without executing the upgrade sequence, thus allowing users to manually rectify the problem.---> 16:35:55 ? ? Creating user specified by MYSQL_USER (user) ...---> 16:35:55 ? ? Creating database opdb ...---> 16:35:55 ? ? Granting privileges to user user for opdb ...---> 16:35:55 ? ? Initialization finished=> sourcing 40-datadir-action.sh ...---> 16:35:55 ? ? Running datadir action: upgrade-warn---> 16:35:55 ? ? Warning: Version of the data could not be determined. It is because the file mysql_upgrade_info is missing in the data directory, which is most probably because it was not created when initialization of data directory. In order to allow seamless updates to the next higher version in the future, the file mysql_upgrade_info will be created. If the data directory was created with a different version than 8.0, it is required to run this container with the MYSQL_DATADIR_ACTION environment variable set to 'force', or run 'mysql_upgrade' utility manually; the mysql_upgrade tool checks the tables and creates such a file as well. For upstream documentation about upgrading, see: https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous-series.html---> 16:35:55 ? ? Storing version '8.0.17' information into the data dir '/var/lib/mysql/data/mysql_upgrade_info'=> sourcing 50-passwd-change.sh ...---> 16:35:55 ? ? Setting passwords ...---> 16:35:55 ? ? Shutting down MySQL ...2020-03-27T16:35:55.555498Z 15 [System] [MY-013172] [Server] Received SHUTDOWN from user root. Shutting down mysqld (Version: 8.0.17).2020-03-27T16:35:57.798891Z 0 [System] [MY-010910] [Server] /opt/rh/rh-mysql80/root/usr/libexec/mysqld: Shutdown complete (mysqld 8.0.17) ?Source distribution.---> 16:35:58 ? ? Cleaning up environment variables MYSQL_USER, MYSQL_PASSWORD, MYSQL_DATABASE and MYSQL_ROOT_PASSWORD ...---> 16:35:58 ? ? Running final exec -- Only MySQL server logs after this point2020-03-27T16:35:58.909778Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.2020-03-27T16:35:58.911732Z 0 [System] [MY-010116] [Server] /opt/rh/rh-mysql80/root/usr/libexec/mysqld (mysqld 8.0.17) starting as process 12020-03-27T16:35:59.321049Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.2020-03-27T16:35:59.353887Z 0 [System] [MY-010931] [Server] /opt/rh/rh-mysql80/root/usr/libexec/mysqld: ready for connections. Version: '8.0.17' ?socket: '/var/lib/mysql/mysql.sock' ?port: 3306 ?Source distribution.2020-03-27T16:35:59.460547Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/lib/mysql/mysqlx.sock' bind-address: '::' port: 33060

對比git的源碼內容gitclone后進行查看比對，可以從日志中看到了定制化腳本被執行：

[root@lb.davidwei-cluster1 /]# git clone https://github.com/sclorg/mysql-containerCloning into 'mysql-container'...[root@lb.davidwei-cluster1 /]# cd mysql-container/[root@lb.davidwei-cluster1 /mysql-container]# ls5.5 ?5.6 ?5.7 ?8.0 ?common ?examples ?imagestreams ?LICENSE ?Makefile ?README.md ?root-common ?s2i-common ?test[root@lb.davidwei-cluster1 /mysql-container]# find ./ -name 50*./5.6/root-common/usr/share/container-scripts/mysql/cnf/50-my-tuning.cnf./5.6/root-common/usr/share/container-scripts/mysql/init/50-passwd-change.sh./root-common/usr/share/container-scripts/mysql/cnf/50-my-tuning.cnf./root-common/usr/share/container-scripts/mysql/init/50-passwd-change.sh

MySQL部署成功后如下圖所示：

[root@lb.davidwei-cluster1 ~]# oc get svcNAME ? ? ? ? ? ? TYPE ? ? ? ?CLUSTER-IP ? ? ?EXTERNAL-IP ? PORT(S) ? ?AGEmy-mysql-rhel7 ? ClusterIP ? 172.30.99.167 ? ? ? ? ?3306/TCP ? 20m[root@lb.davidwei-cluster1 ~]# oc rsh my-mysql-rhel7-1-zcwpcsh-4.2$ mysql -h 172.30.99.167 ?-P 3306 -u user -ppassmysql: [Warning] Using a password on the command line interface can be insecure.Welcome to the MySQL monitor. ?Commands end with ; or \g.Your MySQL connection id is 15Server?version:?8.0.17?Source?distributioCopyright?(c)?2000,?2019,?Oracle?and/or?its?affiliates.?All?rights?reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type?'help;'?or?'\h'?for?help.?Type?'\c'?to?clear?the?current?input?statement.mysql>

截止到目前，我們完成了在OpenShift中部署mysql單實例。

在OpenShift中部署MySQL復制

使用mysql主備復制模板進行部署。這是一個一主一從復制的模板。可以基于這個模板進行定制化。本文不做定制化修改：

(https://github.com/sclorg/mysql-container/blob/master/examples/replica/mysql_replica.json)

[root@lb.davidwei-cluster1 ~]# oc create -f mysql.jsontemplate.template.openshift.io/mysql-replication-example created[root@lb.davidwei-cluster1 ~]# oc get templateNAME ? ? ? ? ? ? ? ? ? ? ? ?DESCRIPTION ? ? ? ? ? ? ? ? PARAMETERS ? ? ? ?OBJECTSmysql-replication-example ? MySQL Replication Example ? 8 (3 generated) ? 6

[root@lb.davidwei-cluster1?~]#?oc?get?podsNAME??????????????????????READY???STATUS??????????????RESTARTS???AGEmy-mysql-rhel7-1-zcwpc????1/1?????Running?????????????0??????????42mmysql-master-2-deploy?????1/1?????Running?????????????0??????????55smysql-master-2-zpvr5??????0/1?????ContainerCreating???0??????????47smysql-slave-2-deploy??????1/1?????Running?????????????0??????????49smysql-slave-2-prlmq???????0/1?????ContainerCreating???0??????????41smysql> SHOW MASTER STATUS;

+------------------+----------+--------------+------------------+

+------------------+----------+--------------+------------------+

| mysql-bin.000002 | ? ? ?107 | foo ? ? ? ? ?| ? ? ? ? ? ? ? ? ?|

+------------------+----------+--------------+------------------+

1 row in set (0.00 sec)mysql> SHOW SLAVE HOSTS;

+------------+-------------+------+------------+

+------------+-------------+------+------------+

| 3314680171 | 172.17.0.20 | 3306 | 1301393349 |

| 3532875540 | 172.17.0.18 | 3306 | 1301393349 |

+------------+-------------+------+------------+

2 rows in set (0.01 sec)

總結

以上是生活随笔為你收集整理的mysql中shift h_MySQL复制技术对比与容器化探究的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。

上一篇： hive向mysql导数据_导入Hive
下一篇： ofbiz mysql 中文安装_ofb