Harbor高可用集群配置
實現方式
復制同步
多harbor實例共享后端存儲
主從復制同步
harbor官方默認提供主從復制的方案來解決鏡像同步問題，通過復制的方式，我們可以實時將測試環境harbor倉庫的鏡像同步到生產環境harbor，類似于如下流程：
在實際生產運維的中，往往需要把鏡像發布到幾十或上百臺集群節點上。這時，單個Registry已經無法滿足大量節點的下載需求，因此要配置多個Registry實例做負載均衡。手工維護多個Registry實例上的鏡像，將是十分繁瑣的事情。Harbor可以支持一主多從的鏡像發布模式，可以解決大規模鏡像發布的難題：
只要往一臺Registry上發布，鏡像就像“仙女散花”般地同步到多個Registry中，高效可靠。
如果是地域分布較廣的集群，還可以采用層次型發布方式，如從集團總部同步到省公司，從省公司再同步到市公司：
然而單靠主從同步，仍然解決不了harbor主節點的單點問題。
雙主復制同步
所謂的雙主復制其實就是復用主從同步實現兩個harbor節點之間的雙向同步，來保證數據的一致性，然后在兩臺harbor前端頂一個負載均衡器將進來的請求分流到不同的實例中去，只要有一個實例中有了新的鏡像，就是自動的同步復制到另外的的實例中去，這樣實現了負載均衡，也避免了單點故障，在一定程度上實現了Harbor的高可用性：
這個方案有一個問題就是有可能兩個Harbor實例中的數據不一致。假設如果一個實例A掛掉了，這個時候有新的鏡像進來，那么新的鏡像就會在另外一個實例B中，后面即使恢復了掛掉的A實例，Harbor實例B也不會自動去同步鏡像，這樣只能手動的先關掉Harbor實例B的復制策略，然后再開啟復制策略，才能讓實例B數據同步，讓兩個實例的數據一致。
在實際生產使用中，主從復制十分的不靠譜。
所以這里推薦使用下面要說的這種方案。
多harbor實例共享后端存儲
共享后端存儲算是一種比較標準的方案，就是多個Harbor實例共享同一個后端存儲，任何一個實例持久化到存儲的鏡像，都可被其他實例中讀取。通過前置LB進來的請求，可以分流到不同的實例中去處理，這樣就實現了負載均衡，也避免了單點故障：
這個方案在實際生產環境中部署需要考慮三個問題：
共享存儲的選取，Harbor的后端存儲目前支持AWS S3、Openstack Swift, Ceph等，在我們的實驗環境里，就直接使用nfs
Session在不同的實例上共享，這個現在其實已經不是問題了，在最新的harbor中，默認session會存放在redis中，我們只需要將redis獨立出來即可。可以通過redis sentinel或者redis cluster等方式來保證redis的可用性。在我們的實驗環境里，仍然使用單臺redis
Harbor多實例數據庫問題，這個也只需要將harbor中的數據庫拆出來獨立部署即可。讓多實例共用一個外部數據庫，數據庫的高可用也可以通過數據庫的高可用方案保證。
環境說明
實驗環境
Ip | Role |
172.16.251.131 | harbor01.dc.com |
172.16.251.132 | harbor02.dc.com |
172.16.251.133 | Storage.dc.com |
172.16.251.134 | Lvs+keepalived |
172.16.251.135 | Lvs+keepalived |
配置說明
安裝配置NFS
1、安裝NFS
root@storage:~# apt install nfs-kernel-server nfs-common
Reading package lists... Done
Building dependency tree??????
Reading state information... Done
The following additional packages will be installed:
? keyutils libnfsidmap2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libtirpc1 python python-minimal python2.7 python2.7-minimal rpcbind
Suggested packages:
? watchdog python-doc python-tk python2.7-doc binutils binfmt-support
The following NEW packages will be installed:
? keyutils libnfsidmap2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libtirpc1 nfs-common nfs-kernel-server python python-minimal python2.7
? python2.7-minimal rpcbind
0 upgraded, 13 newly installed, 0 to remove and 4 not upgraded.
Need to get 4,344 kB of archives.
After this operation, 18.4 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirrors.aliyun.com/ubuntu xenial/main amd64 libnfsidmap2 amd64 0.25-5 [32.2 kB]
Get:2 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libpython2.7-minimal amd64 2.7.12-1ubuntu0~16.04.4 [339 kB]
Get:3 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 python2.7-minimal amd64 2.7.12-1ubuntu0~16.04.4 [1,261 kB]
Get:4 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 python-minimal amd64 2.7.12-1~16.04 [28.1 kB]
Get:5 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libpython2.7-stdlib amd64 2.7.12-1ubuntu0~16.04.4 [1,880 kB]
Get:6 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 python2.7 amd64 2.7.12-1ubuntu0~16.04.4 [224 kB]
Get:7 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libpython-stdlib amd64 2.7.12-1~16.04 [7,768 B]
Get:8 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 python amd64 2.7.12-1~16.04 [137 kB]
Get:9 http://mirrors.aliyun.com/ubuntu xenial/main amd64 keyutils amd64 1.5.9-8ubuntu1 [47.1 kB]
Get:10 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libtirpc1 amd64 0.2.5-1ubuntu0.1 [75.4 kB]
Get:11 http://mirrors.aliyun.com/ubuntu xenial/main amd64 rpcbind amd64 0.2.3-0.2 [40.3 kB]
Get:12 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 nfs-common amd64 1:1.2.8-9ubuntu12.1 [184 kB]
Get:13 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 nfs-kernel-server amd64 1:1.2.8-9ubuntu12.1 [88.0 kB]
Fetched 4,344 kB in 0s (5,046 kB/s)?????????
Selecting previously unselected package libnfsidmap2:amd64.
(Reading database ... 60656 files and directories currently installed.)
Preparing to unpack .../libnfsidmap2_0.25-5_amd64.deb ...
Unpacking libnfsidmap2:amd64 (0.25-5) ...
Selecting previously unselected package libpython2.7-minimal:amd64.
Preparing to unpack .../libpython2.7-minimal_2.7.12-1ubuntu0~16.04.4_amd64.deb ...
Unpacking libpython2.7-minimal:amd64 (2.7.12-1ubuntu0~16.04.4) ...
Selecting previously unselected package python2.7-minimal.
Preparing to unpack .../python2.7-minimal_2.7.12-1ubuntu0~16.04.4_amd64.deb ...
Unpacking python2.7-minimal (2.7.12-1ubuntu0~16.04.4) ...
Selecting previously unselected package python-minimal.
Preparing to unpack .../python-minimal_2.7.12-1~16.04_amd64.deb ...
Unpacking python-minimal (2.7.12-1~16.04) ...
Selecting previously unselected package libpython2.7-stdlib:amd64.
Preparing to unpack .../libpython2.7-stdlib_2.7.12-1ubuntu0~16.04.4_amd64.deb ...
Unpacking libpython2.7-stdlib:amd64 (2.7.12-1ubuntu0~16.04.4) ...
Selecting previously unselected package python2.7.
Preparing to unpack .../python2.7_2.7.12-1ubuntu0~16.04.4_amd64.deb ...
Unpacking python2.7 (2.7.12-1ubuntu0~16.04.4) ...
Selecting previously unselected package libpython-stdlib:amd64.
Preparing to unpack .../libpython-stdlib_2.7.12-1~16.04_amd64.deb ...
Unpacking libpython-stdlib:amd64 (2.7.12-1~16.04) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Setting up libpython2.7-minimal:amd64 (2.7.12-1ubuntu0~16.04.4) ...
Setting up python2.7-minimal (2.7.12-1ubuntu0~16.04.4) ...
Linking and byte-compiling packages for runtime python2.7...
Setting up python-minimal (2.7.12-1~16.04) ...
Selecting previously unselected package python.
(Reading database ... 61412 files and directories currently installed.)
Preparing to unpack .../python_2.7.12-1~16.04_amd64.deb ...
Unpacking python (2.7.12-1~16.04) ...
Selecting previously unselected package keyutils.
Preparing to unpack .../keyutils_1.5.9-8ubuntu1_amd64.deb ...
Unpacking keyutils (1.5.9-8ubuntu1) ...
Selecting previously unselected package libtirpc1:amd64.
Preparing to unpack .../libtirpc1_0.2.5-1ubuntu0.1_amd64.deb ...
Unpacking libtirpc1:amd64 (0.2.5-1ubuntu0.1) ...
Selecting previously unselected package rpcbind.
Preparing to unpack .../rpcbind_0.2.3-0.2_amd64.deb ...
Unpacking rpcbind (0.2.3-0.2) ...
Selecting previously unselected package nfs-common.
Preparing to unpack .../nfs-common_1%3a1.2.8-9ubuntu12.1_amd64.deb ...
Unpacking nfs-common (1:1.2.8-9ubuntu12.1) ...
Selecting previously unselected package nfs-kernel-server.
Preparing to unpack .../nfs-kernel-server_1%3a1.2.8-9ubuntu12.1_amd64.deb ...
Unpacking nfs-kernel-server (1:1.2.8-9ubuntu12.1) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21.15) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up libnfsidmap2:amd64 (0.25-5) ...
Setting up libpython2.7-stdlib:amd64 (2.7.12-1ubuntu0~16.04.4) ...
Setting up python2.7 (2.7.12-1ubuntu0~16.04.4) ...
Setting up libpython-stdlib:amd64 (2.7.12-1~16.04) ...
Setting up python (2.7.12-1~16.04) ...
Setting up keyutils (1.5.9-8ubuntu1) ...
Setting up libtirpc1:amd64 (0.2.5-1ubuntu0.1) ...
Setting up rpcbind (0.2.3-0.2) ...
Setting up nfs-common (1:1.2.8-9ubuntu12.1) ...
?
Creating config file /etc/idmapd.conf with new version
?
Creating config file /etc/default/nfs-common with new version
Adding system user `statd' (UID 111) ...
Adding new user `statd' (UID 111) with group `nogroup' ...
Not creating home directory `/var/lib/nfs'.
nfs-utils.service is a disabled or a static unit, not starting it.
Setting up nfs-kernel-server (1:1.2.8-9ubuntu12.1) ...
?
Creating config file /etc/exports with new version
?
Creating config file /etc/default/nfs-kernel-server with new version
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21.15) ...
Processing triggers for ureadahead (0.100.0-19) ...
?
2、配置NFS
root@storage:~# mkdir /data
root@storage:~# vim /etc/exports
root@storage:~/redis-mysql# cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
#????? to NFS clients.? See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes?????? hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4??????? gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes? gss/krb5i(rw,sync,no_subtree_check)
#
/data?? *(rw,no_root_squash)
root@storage:~# chmod 777 -R /data/
root@storage:~# systemctl restart nfs-server.service
root@storage:~# showmount -e
Export list for storage.dc.com:
/data *
?
安裝redis和mysql
這里我們就直接通過docker安裝，docker-compose.yml文件內容如下：
root@storage:~/redis-mysql# cat docker-compose.yml
version: '3'
services:
? mysql-server:
??? hostname: mysql-server
??? container_name: mysql-server
??? image: mysql:5.7
??? network_mode: host
??? volumes:
????? - /mysql57/data:/var/lib/mysql
??? command: --character-set-server=utf8
??? environment:
????? MYSQL_ROOT_PASSWORD: 123456
? redis:
??? hostname: redis-server
??? container_name: redis-server
??? image: redis:3
network_mode: host
?
啟動運行
?
root@storage:~/redis-mysql# docker-compose up -d
Pulling mysql-server (mysql:5.7)...
5.7: Pulling from library/mysql
177e7ef0df69: Pull complete
cac25352c4c8: Pull complete
8585afabb40a: Pull complete
1e4af4996053: Pull complete
c326522894da: Pull complete
9020d6b6b171: Pull complete
55eb37ec6e5f: Pull complete
1a9d2f77e0e7: Pull complete
d7e648ad64aa: Pull complete
4120d828ea6b: Pull complete
3b39dc5451af: Pull complete
Pulling redis (redis:3)...
3: Pulling from library/redis
f17d81b4b692: Pull complete
b32474098757: Pull complete
8980cabe8bc2: Pull complete
58af19693e78: Pull complete
a977782cf22d: Pull complete
9c1e268980b7: Pull complete
Creating redis-server ... done
Creating mysql-server ... done
?
root@storage:~/redis-mysql# docker ps -a
CONTAINER ID??????? IMAGE?????????????? COMMAND????????????????? CREATED???????????? STATUS????????????? PORTS?????????????? NAMES
e6533520c245??????? mysql:5.7?????????? "docker-entrypoint.s…"?? 28 minutes ago????? Up 28 minutes?????????????????????????? mysql-server
a26c846816b0??????? redis:3???????????? "docker-entrypoint.s…"?? 28 minutes ago????? Up 28 minutes?????????????????????????? redis-server
?
導入registry數據庫
配置好了mysql以后，還需要往mysql數據庫中導入harbor registry庫。安裝了一個單機版harbor，啟動了一個mysql，里面有一個registry數據庫，直接導出來，然后再導入到新數據庫中：
# 導出數據庫：
?
docker exec -it harbor_db /bin/bash
mysqldump -uroot -p --databases registry > registry.dump
?
# 在宿主機上將registry.dump復制出來
docker cp? harbor_db:/registry.dump ./
?
# 將宿主機上的registry.dump復制到獨立的mysql容器中
?
docker cp ./registry.dump <mysql-server-container>:/registry.dump
?
# 在獨立的mysql容器將將registry數據庫導入
?
docker exec -it <mysql-server-container> /bin/bash
?
mysql -uroot -p
?
mysql> source /registry.dump
?
配置harbor
掛載NFS目錄
root@harbor01:~# showmount -e 172.16.251.133
Export list for 172.16.251.133:
/data *
root@harbor01:~# mkdir /data
root@harbor01:~# mount 172.16.251.133:/data /data
root@harbor01:~# df -h
Filesystem??????????? Size? Used Avail Use% Mounted on
udev????????????????? 7.9G???? 0? 7.9G?? 0% /dev
tmpfs???????????????? 1.6G? 8.9M? 1.6G?? 1% /run
/dev/sda3????????????? 42G? 2.0G?? 38G?? 5% /
tmpfs???????????????? 7.9G???? 0? 7.9G?? 0% /dev/shm
tmpfs???????????????? 5.0M???? 0? 5.0M?? 0% /run/lock
tmpfs???????????????? 7.9G???? 0? 7.9G?? 0% /sys/fs/cgroup
/dev/sda1???????????? 464M?? 59M? 377M? 14% /boot
tmpfs???????????????? 1.6G???? 0? 1.6G?? 0% /run/user/0
172.16.251.133:/data?? 18G? 2.6G?? 15G? 16% /data
?
生成證書
root@harbor01:~# openssl genrsa -des3 -out server.key 1024
Generating RSA private key, 1024 bit long modulus
............++++++
..................++++++
e is 65537 (0x10001)
Enter pass phrase for server.key:
Verifying - Enter pass phrase for server.key:
?
root@harbor01:~# openssl rsa -in server.key -out server.key
Enter pass phrase for server.key:
?
root@harbor01:~# openssl req -new -key server.key -out server.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CN
State or Province Name (full name) [Some-State]:Shanghai
Locality Name (eg, city) []:Pudong
Organization Name (eg, company) [Internet Widgits Pty Ltd]:BL
Organizational Unit Name (eg, section) []:IT
Common Name (e.g. server FQDN or YOUR name) []:harbor.dc.com
Email Address []:root@harbor.dc.com
?
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
?
root@harbor01:~# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
Signature ok
subject=/C=CN/ST=Shanghai/L=Pudong/O=BL/OU=IT/CN=harbor.dc.com/emailAddress=root@harbor.dc.com
Getting Private key
?
配置harbor.cfg
解壓harbor-offline-installer-v1.5.2.tgz進入harbor目錄
root@harbor01:~/harbor# cat harbor.cfg |grep -v '^$' |grep -v '^#'
_version = 1.5.0
hostname = harbor.dc.com
ui_url_protocol = https
max_job_workers = 50
customize_crt = on
ssl_cert = /data/certs/server.crt
ssl_cert_key = /data/certs/server.key
secretkey_path = /data
admiral_url = NA
log_rotate_count = 50
log_rotate_size = 200M
http_proxy =
https_proxy =
no_proxy = 127.0.0.1,localhost,ui
email_identity =
email_server = smtp.mydomain.com
email_server_port = 25
email_username = sample_admin@mydomain.com
email_password = abc
email_from = admin <sample_admin@mydomain.com>
email_ssl = false
email_insecure = false
harbor_admin_password = Harbor12345
auth_mode = db_auth
ldap_url = ldaps://ldap.mydomain.com
ldap_basedn = ou=people,dc=mydomain,dc=com
ldap_uid = uid
ldap_scope = 2
ldap_timeout = 5
ldap_verify_cert = true
ldap_group_basedn = ou=group,dc=mydomain,dc=com
ldap_group_filter = objectclass=group
ldap_group_gid = cn
ldap_group_scope = 2
self_registration = on
token_expiration = 30
project_creation_restriction = everyone
db_host = 172.16.251.133
db_password = 123456
db_port = 3306
db_user = root
redis_url = 172.16.251.133:6379
clair_db_host = postgres
clair_db_password = password
clair_db_port = 5432
clair_db_username = postgres
clair_db = postgres
uaa_endpoint = uaa.mydomain.org
uaa_clientid = id
uaa_clientsecret = secret
uaa_verify_cert = true
uaa_ca_cert = /path/to/ca.pem
registry_storage_provider_name = filesystem
registry_storage_provider_config =
?
部署harbor
root@harbor01:~/harbor# cp ha/docker-compose.yml .
root@harbor01:~/harbor# ./prepare
Clearing the configuration file: ./common/config/jobservice/config.yml
Clearing the configuration file: ./common/config/jobservice/env
Clearing the configuration file: ./common/config/db/env
Clearing the configuration file: ./common/config/adminserver/env
Clearing the configuration file: ./common/config/ui/app.conf
Clearing the configuration file: ./common/config/ui/env
Clearing the configuration file: ./common/config/ui/private_key.pem
Clearing the configuration file: ./common/config/registry/root.crt
Clearing the configuration file: ./common/config/registry/config.yml
Clearing the configuration file: ./common/config/log/logrotate.conf
Clearing the configuration file: ./common/config/nginx/cert/server.key
Clearing the configuration file: ./common/config/nginx/cert/server.crt
Clearing the configuration file: ./common/config/nginx/nginx.conf
loaded secret from file: /data/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/ui/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/log/logrotate.conf
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/ui/app.conf
Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.
?
root@harbor01:~/harbor# ./install.sh
?
[Step 0]: checking installation environment ...
?
Note: docker version: 18.09.0
?
Note: docker-compose version: 1.23.2
?
[Step 1]: loading Harbor images ...
Loaded image: vmware/registry-photon:v2.6.2-v1.5.2
Loaded image: vmware/photon:1.0
b9a9313d1f80: Loading layer [==================================================>]? 457.3MB/457.3MB
a17318dff4d5: Loading layer [==================================================>]? 9.216kB/9.216kB
11b0dc760d85: Loading layer [==================================================>]? 9.216kB/9.216kB
ab253bfa0789: Loading layer [==================================================>]?? 7.68kB/7.68kB
7c41921b687e: Loading layer [==================================================>]? 1.536kB/1.536kB
Loaded image: vmware/mariadb-photon:v1.5.2
Loaded image: vmware/harbor-log:v1.5.2
6c8870489188: Loading layer [==================================================>]? 61.24MB/61.24MB
Loaded image: vmware/nginx-photon:v1.5.2
d6e1f5663df8: Loading layer [==================================================>]? 79.23MB/79.23MB
1e7414733b92: Loading layer [==================================================>]? 10.95MB/10.95MB
740974ce05ad: Loading layer [==================================================>]?? 17.3MB/17.3MB
8e5664ede767: Loading layer [==================================================>]? 15.87kB/15.87kB
86e94116f829: Loading layer [==================================================>]? 3.072kB/3.072kB
173729674517: Loading layer [==================================================>]? 28.24MB/28.24MB
Loaded image: vmware/notary-signer-photon:v0.5.1-v1.5.2
22c53afb5f7e: Loading layer [==================================================>]??? 151MB/151MB
2b4cfcba43e3: Loading layer [==================================================>]? 6.656kB/6.656kB
11d44832002d: Loading layer [==================================================>]? 2.048kB/2.048kB
f805651773a5: Loading layer [==================================================>]?? 7.68kB/7.68kB
Loaded image: vmware/postgresql-photon:v1.5.2
b60d71996f91: Loading layer [==================================================>]? 11.78kB/11.78kB
9f7d20f3523b: Loading layer [==================================================>]?? 2.56kB/2.56kB
1928f43498e2: Loading layer [==================================================>]? 3.072kB/3.072kB
Loaded image: vmware/harbor-db:v1.5.2
74bdb194304b: Loading layer [==================================================>]? 79.23MB/79.23MB
97251db4b4f7: Loading layer [==================================================>]? 20.91MB/20.91MB
7c3dce375b3a: Loading layer [==================================================>]? 20.91MB/20.91MB
Loaded image: vmware/harbor-jobservice:v1.5.2
3e7aac7aae19: Loading layer [==================================================>]? 212.2MB/212.2MB
8e7d884dc74c: Loading layer [==================================================>]? 10.93MB/10.93MB
12ed92f1f090: Loading layer [==================================================>]? 2.048kB/2.048kB
81a629ba5751: Loading layer [==================================================>]? 48.13kB/48.13kB
2171b5a262b0: Loading layer [==================================================>]? 10.97MB/10.97MB
Loaded image: vmware/clair-photon:v2.0.4-v1.5.2
b5630b8d91ab: Loading layer [==================================================>]? 79.23MB/79.23MB
89643b845018: Loading layer [==================================================>]? 15.37MB/15.37MB
7b6fa7dbcef0: Loading layer [==================================================>]? 15.37MB/15.37MB
Loaded image: vmware/harbor-adminserver:v1.5.2
bae5fd08868a: Loading layer [==================================================>]? 79.24MB/79.24MB
ca9e737c9888: Loading layer [==================================================>]? 24.41MB/24.41MB
5630a0e582d3: Loading layer [==================================================>]? 7.168kB/7.168kB
68eedf2397cc: Loading layer [==================================================>]? 10.56MB/10.56MB
74ac61aa9a2b: Loading layer [==================================================>]?? 24.4MB/24.4MB
Loaded image: vmware/harbor-ui:v1.5.2
1b5a1160526a: Loading layer [==================================================>]? 136.2MB/136.2MB
24d9672d5b26: Loading layer [==================================================>]? 3.072kB/3.072kB
fe4d52841f3d: Loading layer [==================================================>]?? 59.9kB/59.9kB
bffe30bd1941: Loading layer [==================================================>]? 61.95kB/61.95kB
Loaded image: vmware/redis-photon:v1.5.2
dccbbe2d4eff: Loading layer [==================================================>]? 12.16MB/12.16MB
d1291eaee2a3: Loading layer [==================================================>]?? 17.3MB/17.3MB
610d6a304e63: Loading layer [==================================================>]? 15.87kB/15.87kB
2dabac4548fb: Loading layer [==================================================>]? 3.072kB/3.072kB
46189122932e: Loading layer [==================================================>]? 29.46MB/29.46MB
Loaded image: vmware/notary-server-photon:v0.5.1-v1.5.2
454c81edbd3b: Loading layer [==================================================>]? 135.2MB/135.2MB
e99db1275091: Loading layer [==================================================>]? 395.4MB/395.4MB
051e4ee23882: Loading layer [==================================================>]? 9.216kB/9.216kB
6cca4437b6f6: Loading layer [==================================================>]? 9.216kB/9.216kB
1d48fc08c8bc: Loading layer [==================================================>]?? 7.68kB/7.68kB
0419724fd942: Loading layer [==================================================>]? 1.536kB/1.536kB
543c0c1ee18d: Loading layer [==================================================>]? 655.2MB/655.2MB
4190aa7e89b8: Loading layer [==================================================>]? 103.9kB/103.9kB
Loaded image: vmware/harbor-migrator:v1.5.0
?
?
[Step 2]: preparing environment ...
Clearing the configuration file: ./common/config/jobservice/config.yml
Clearing the configuration file: ./common/config/jobservice/env
Clearing the configuration file: ./common/config/db/env
Clearing the configuration file: ./common/config/adminserver/env
Clearing the configuration file: ./common/config/ui/app.conf
Clearing the configuration file: ./common/config/ui/env
Clearing the configuration file: ./common/config/ui/private_key.pem
Clearing the configuration file: ./common/config/registry/root.crt
Clearing the configuration file: ./common/config/registry/config.yml
Clearing the configuration file: ./common/config/log/logrotate.conf
Clearing the configuration file: ./common/config/nginx/cert/server.key
Clearing the configuration file: ./common/config/nginx/cert/server.crt
Clearing the configuration file: ./common/config/nginx/nginx.conf
loaded secret from file: /data/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/ui/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/log/logrotate.conf
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/ui/app.conf
Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.
?
?
[Step 3]: checking existing instance of Harbor ...
?
?
[Step 4]: starting Harbor ...
Creating harbor-log ... done
Creating registry?????????? ... done
Creating harbor-adminserver ... done
Creating harbor-ui????????? ... done
Creating nginx????????????? ... done
Creating harbor-jobservice? ... done
?
? ----Harbor has been installed and started successfully.----
?
Now you should be able to visit the admin portal at https://harbor01.dc.com.
For more details, please visit https://github.com/vmware/harbor .
?
root@harbor01:~/harbor# docker ps -a
CONTAINER ID??????? IMAGE?????????? ???????????????????????COMMAND????????????????? CREATED???????????? STATUS???????????????????????????? PORTS????????????????????????????????????????????????????????????? NAMES
cdf3685c2d94??????? vmware/harbor-jobservice:v1.5.2??????? "/harbor/start.sh"?? ????13 seconds ago????? Up 12 seconds???????????????????????????????????????????????????????????????????????????????????????? harbor-jobservice
5de16bad4f2d??????? vmware/nginx-photon:v1.5.2???????????? "nginx -g 'daemon of…"?? 13 seconds ago????? Up 12 seconds (health: starting)?? 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp?? nginx
15406c90e365??????? vmware/harbor-ui:v1.5.2??????????????? "/harbor/start.sh"?????? 14 seconds ago????? Up 13 seconds (health: starting)??????????????????? ??????????????????????????????????????????????????harbor-ui
9e7330ce8130??????? vmware/harbor-adminserver:v1.5.2?????? "/harbor/start.sh"?????? 14 seconds ago????? Up 13 seconds (health: starting)??????????????????????????????????????????????????????????? ??????????harbor-adminserver
d344a7ae027e??????? vmware/registry-photon:v2.6.2-v1.5.2?? "/entrypoint.sh serv…"?? 14 seconds ago????? Up 13 seconds (health: starting)?? 5000/tcp?????????????????????????????????????????????????????????? registry
3c84cf5271f6??????? vmware/harbor-log:v1.5.2?????????????? "/bin/sh -c /usr/loc…"?? 15 seconds ago????? Up 14 seconds (health: starting)?? 127.0.0.1:1514->10514/tcp??????????????????????? ??????????????????harbor-log
root@harbor01:~/harbor#
?
harbor測試
?
安裝配置LVS
root@Userver05:~# apt install ipvsadm keepalived
Reading package lists... Done
Building dependency tree??????
Reading state information... Done
The following additional packages will be installed:
? libnl-route-3-200 libsensors4 libsnmp-base libsnmp30
Suggested packages:
? heartbeat ldirectord lm-sensors snmp-mibs-downloader
The following NEW packages will be installed:
? ipvsadm keepalived libnl-route-3-200 libsensors4 libsnmp-base libsnmp30
0 upgraded, 6 newly installed, 0 to remove and 9 not upgraded.
Need to get 1,429 kB of archives.
After this operation, 5,570 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libnl-route-3-200 amd64 3.2.27-1ubuntu0.16.04.1 [124 kB]
Get:2 http://mirrors.aliyun.com/ubuntu xenial/main amd64 libsensors4 amd64 1:3.4.0-2 [28.4 kB]
Get:3 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libsnmp-base all 5.7.3+dfsg-1ubuntu4.2 [224 kB]
Get:4 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libsnmp30 amd64 5.7.3+dfsg-1ubuntu4.2 [813 kB]
Get:5 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 keepalived amd64 1:1.2.24-1ubuntu0.16.04.1 [205 kB]
Get:6 http://mirrors.aliyun.com/ubuntu xenial/main amd64 ipvsadm amd64 1:1.28-3 [35.5 kB]
Fetched 1,429 kB in 0s (2,824 kB/s)
Selecting previously unselected package libnl-route-3-200:amd64.
(Reading database ... 60370 files and directories currently installed.)
Preparing to unpack .../libnl-route-3-200_3.2.27-1ubuntu0.16.04.1_amd64.deb ...
Unpacking libnl-route-3-200:amd64 (3.2.27-1ubuntu0.16.04.1) ...
Selecting previously unselected package libsensors4:amd64.
Preparing to unpack .../libsensors4_1%3a3.4.0-2_amd64.deb ...
Unpacking libsensors4:amd64 (1:3.4.0-2) ...
Selecting previously unselected package libsnmp-base.
Preparing to unpack .../libsnmp-base_5.7.3+dfsg-1ubuntu4.2_all.deb ...
Unpacking libsnmp-base (5.7.3+dfsg-1ubuntu4.2) ...
Selecting previously unselected package libsnmp30:amd64.
Preparing to unpack .../libsnmp30_5.7.3+dfsg-1ubuntu4.2_amd64.deb ...
Unpacking libsnmp30:amd64 (5.7.3+dfsg-1ubuntu4.2) ...
Selecting previously unselected package keepalived.
Preparing to unpack .../keepalived_1%3a1.2.24-1ubuntu0.16.04.1_amd64.deb ...
Unpacking keepalived (1:1.2.24-1ubuntu0.16.04.1) ...
Selecting previously unselected package ipvsadm.
Preparing to unpack .../ipvsadm_1%3a1.28-3_amd64.deb ...
Unpacking ipvsadm (1:1.28-3) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu21.15) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up libnl-route-3-200:amd64 (3.2.27-1ubuntu0.16.04.1) ...
Setting up libsensors4:amd64 (1:3.4.0-2) ...
Setting up libsnmp-base (5.7.3+dfsg-1ubuntu4.2) ...
Setting up libsnmp30:amd64 (5.7.3+dfsg-1ubuntu4.2) ...
Setting up keepalived (1:1.2.24-1ubuntu0.16.04.1) ...
Setting up ipvsadm (1:1.28-3) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21.15) ...
Processing triggers for ureadahead (0.100.0-19) ...
?
root@Userver05:~# cat /etc/keepalived/keepalived.conf
global_defs {
? router_id haborlb
}
vrrp_sync_groups VG1 {
? group {
??? VI_1
? }
}
#Please change "ens160" to the interface name on you loadbalancer hosts.
#In some case it will be eth0, ens16xxx etc.
vrrp_instance VI_1 {
? interface ens32
?
? track_interface {
??? ens32
? }
?
? state BACKUP
? virtual_router_id 51
? priority 8
?
? virtual_ipaddress {
??? 172.16.251.136/32
? }
? advert_int 1
? authentication {
??? auth_type PASS
??? auth_pass d0cker
? }
?
}
########## Section for Harbor use HTTP protocol ######################
#Please change <change_to_VIP_address>, <harbor_node1_ip>, <harbor_node2_ip> to real ip address
#virtual_server <change_to_VIP_address> 80 {
#? delay_loop 15
#? lb_algo rr
#? lb_kind DR
#? protocol TCP
#? nat_mask 255.255.255.0
#? persistence_timeout 10
#
#? real_server <harbor_node1_ip> 80 {
#??? weight 10
#??? MISC_CHECK {
#??????? misc_path "/usr/local/bin/check.sh <harbor_node1_ip>"
#??????? misc_timeout 5
#??? }
#? }
#
#? real_server <harbor_node2_ip> 80 {
#??? weight 10
#??? MISC_CHECK {
#??????? misc_path "/usr/local/bin/check.sh <harbor_node2_ip>"
#??????? misc_timeout 5
#??? }
#? }
#}
?
#########################End of HTTP############################
?
##########################HTTPS#################################
#Please uncomment the follow when harbor running under https
virtual_server 172.16.251.136 443 {
? delay_loop 15
? lb_algo rr
? lb_kind DR
? protocol TCP
? nat_mask 255.255.255.0
? persistence_timeout 10
?
? real_server 172.16.251.131 443 {
??? weight 10
??? MISC_CHECK {
?????? misc_path "/usr/local/bin/check.sh 172.16.251.131"
?????? misc_timeout 5
??? }
? }
?
? real_server 172.16.251.132 443 {
??? weight 10
??? MISC_CHECK {
?????? misc_path "/usr/local/bin/check.sh 172.16.251.132"
?????? misc_timeout 5
??? }
? }
}
#########################End of HTTPS Section#################
?
?
root@Userver05:~# cat /usr/local/bin/check.sh
#!/bin/bash
?
set -e
#get protocol
?
#LOG=/var/log/keepalived_check.log
nodeip=$1
nodeaddress="http://${nodeip}"
http_code=`curl -s -o /dev/null -w "%{http_code}" ${nodeaddress}`
?
if [ $http_code == 200 ] ; then
? protocol="http"
elif [ $http_code == 301 ]
then
? protocol="https"
else
#? echo "`date +"%Y-%m-%d %H:%M:%S"` $1, CHECK_CODE=$http_code" >> $LOG
? exit 1
fi
?
systeminfo=`curl -k -o - -s ${protocol}://${nodeip}/api/systeminfo`
?
echo $systeminfo | grep "registry_url"
if [ $? != 0 ] ; then
? exit 1
fi
#TODO need to check Clair, but currently Clair status api is unreachable from LB.
# echo $systeminfo | grep "with_clair" | grep "true"
# if [ $? == 0 ] ; then
# clair is enabled
# do some clair check
# else
# clair is disabled
# fi
?
#check top api
?
http_code=`curl -k -s -o /dev/null -w "%{http_code}\n" ${protocol}://${nodeip}/api/repositories/top`
set +e
if [ $http_code == 200 ] ; then
? exit 0
else
? exit 1
fi
?
root@Userver04:~# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
? -> RemoteAddress:Port?????????? Forward Weight ActiveConn InActConn
TCP? 172.16.251.136:https rr persistent 10
? -> 172.16.251.131:https???????? Route?? 10???? 2????????? 0????????
? -> 172.16.251.132:https???????? Route?? 10???? 0????????? 0????????
root@Userver04:~#
?
測試
?
root@Userver04:~# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
? -> RemoteAddress:Port?????????? Forward Weight ActiveConn InActConn
TCP? 172.16.251.136:https rr persistent 10
? -> 172.16.251.132:https???????? Route?? 10???? 6????????? 1????????
root@Userver04:~#
?
轉載于:https://blog.51cto.com/coolsky/2343408
總結
以上是生活随笔為你收集整理的Harbor高可用集群配置的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： iOS个人中心渐变动画、微信对话框、标签
- 下一篇： 块状元素、内联元素和内联块状元素