【 使用环境 】测试环境
【 OB or 其他组件 】
【 使用版本 】 OB 4.0
【问题描述】使用OBD 部署的OB 4.0 BP1 版本三节点集群如何升级到最新的BP2 版本
【复现路径】问题出现前后相关操作
【问题现象及影响】
【附件】
OBD升级流程就可以完成啊
OBD 1.6.2 支持 4.0 BP升级
下载地址：OceanBase 社区
升级命令 obd cluster upgrade
文档地址：https://www.oceanbase.com/docs/community-obd-cn-10000000001031933
可以通过 package_hash 来指定特定的版本。当您使用 obd mirror 相关命令时会打印出组件的 md5 值，这个值即为 package_hash
详参文档：https://www.oceanbase.com/docs/community-obd-cn-10000000001031930
OBD本身升级可以通过obd update
单机环境：
obd:1.6.0升级至1.6.2
ob:4.0.0升级至4.0.0-bp2
注意：本次以离线升级为主！
升级obd
- 在线升级，前提是公网环境
直接使用 obd update 命令升级 OBD即可 - 离线升级，前提本地已上传好obd的新版本包（ob-deploy-1.6.2-10.el7.x86_64.rpm）
– 禁用远程镜像
[root@test001 opt]# obd mirror disable remote
Disable remote ok
– 拷贝新版本obd包
[root@test001 opt]# obd mirror clone /opt/ob-deploy-1.6.2-10.el7.x86_64.rpm
name: ob-deploy
version: 1.6.2
release:10.el7
arch: x86_64
md5: 4542035d1b1af411c93cfe9f1dd78327c2ea05d4
add /opt/ob-deploy-1.6.2-10.el7.x86_64.rpm to local mirror
– 升级obd
[root@test001 opt]# obd update
Found a higher version package for OBD
name: ob-deploy
version: 1.6.2
release:10.el7
arch: x86_64
md5: 4542035d1b1af411c93cfe9f1dd78327c2ea05d4
Upgrade successful.
Current version : 1.6.2
升级oceanbase
- 在线升级，前提是公网环境
可直接使用 obd cluster upgrade 命令升级 OceanBase 数据库 - 离线升级，前提本地已上传好oceanbase所需的新版本rpm包（oceanbase-ce-4.0.0.0-102000032022120718.el7.x86_64.rpm）
– 升级完的obd，远程仓库是打开的，需要再次禁用
[root@test001 opt]# obd mirror disable remote
Disable remote ok
– 拷贝新版本oceanbase包
[root@test001 opt]# obd mirror clone /opt/oceanbase-ce-4.0.0.0-102000032022120718.el7.x86_64.rpm
name: oceanbase-ce
version: 4.0.0.0
release:102000032022120718.el7
arch: x86_64
md5: e1c71a4cb252e51f210aae129cee0f09613d42bb
add /opt/oceanbase-ce-4.0.0.0-102000032022120718.el7.x86_64.rpm to local mirror
– 查看md5值，同版本升级，需要md5值做唯一验证
[root@test001 opt]# obd mirror list local
±---------------------------------------------------------------------------------------------------------+
| local Package List |
±------------------±--------±-----------------------±-------±-----------------------------------------+
| name | version | release | arch | md5 |
±------------------±--------±-----------------------±-------±-----------------------------------------+
| oceanbase-ce | 4.0.0.0 | 100000272022110114.el7 | x86_64 | 42611dc51ca9bb28f36e60e4406ceea4a74914c7 |
| obproxy-ce | 4.0.0 | 5.el7 | x86_64 | de53232a951184fad75b15884458d85e31d2f6c3 |
| oceanbase-ce-libs | 4.0.0.0 | 100000272022110114.el7 | x86_64 | 188919f8128394bf9b62e3989220ded05f1d14da |
| ob-deploy | 1.6.2 | 10.el7 | x86_64 | 4542035d1b1af411c93cfe9f1dd78327c2ea05d4 |
| oceanbase-ce | 4.0.0.0 | 102000032022120718.el7 | x86_64 | e1c71a4cb252e51f210aae129cee0f09613d42bb |
±------------------±--------±-----------------------±-------±-----------------------------------------+
– 升级ob
[root@test001 opt]# obd cluster upgrade lzq -c oceanbase-ce -V 4.0.0.0 --usable e1c71a4cb252e51f210aae129cee0f09613d42bb
Get local repositories and plugins ok
±----------------------------------------------------------------------------------------------------+
| Search oceanbase-ce 4.0.0.0 Result |
±-------------±--------±-----------------------±-------±-----------------------------------------+
| name | version | release | arch | md5 |
±-------------±--------±-----------------------±-------±-----------------------------------------+
| oceanbase-ce | 4.0.0.0 | 100000272022110114.el7 | x86_64 | 42611dc51ca9bb28f36e60e4406ceea4a74914c7 |
| oceanbase-ce | 4.0.0.0 | 102000032022120718.el7 | x86_64 | e1c71a4cb252e51f210aae129cee0f09613d42bb |
±-------------±--------±-----------------------±-------±-----------------------------------------+
install oceanbase-ce-4.0.0.0 for local ok
Open ssh connection ok
Connect to observer ok
±------------------------------------------------------------------------------------------------------------+
| Packages Will Be Used |
±-------------±--------±-----------------------±-------±-----------------------------------------±------+
| name | version | release | arch | md5 | mark |
±-------------±--------±-----------------------±-------±-----------------------------------------±------+
| oceanbase-ce | 4.0.0.0 | 100000272022110114.el7 | x86_64 | 42611dc51ca9bb28f36e60e4406ceea4a74914c7 | start |
| oceanbase-ce | 4.0.0.0 | 102000032022120718.el7 | x86_64 | e1c71a4cb252e51f210aae129cee0f09613d42bb | dest |
±-------------±--------±-----------------------±-------±-----------------------------------------±------+
If you use a non-official release, we cannot guarantee a successful upgrade or technical support when you fail. Make sure that you want to use the above package to upgrade. [y/n]: y
Remote oceanbase-ce-4.0.0.0-102000032022120718.el7-e1c71a4cb252e51f210aae129cee0f09613d42bb repository install ok
Remote oceanbase-ce-4.0.0.0-102000032022120718.el7-e1c71a4cb252e51f210aae129cee0f09613d42bb repository lib check ok
Start observer ok
observer program health check ok
Connect to observer ok
Exec upgrade_checker.py ok
Enable upgrade mode ok
Exec upgrade_pre.py ok
Upgrade ok
Remote oceanbase-ce-4.0.0.0-102000032022120718.el7-e1c71a4cb252e51f210aae129cee0f09613d42bb repository lib check ok
Stop observer ok
Start observer ok
observer program health check ok
Connect to observer ok
Exec upgrade_post.py ok
Disable upgrade mode ok
Root inspection ok
Exec upgrade_post_checker.py ok
Wait for observer init ok
±-------------------------------------------------+
| observer |
±---------------±--------±-----±------±-------+
| ip | version | port | zone | status |
±---------------±--------±-----±------±-------+
| xx.xx.xx.xx | 4.0.0.0 | 2881 | zone1 | ACTIVE |
±---------------±--------±-----±------±-------+
obclient -hxx.xx.xx.xx -P2881 -uroot -Doceanbase -A
升级obproxy
- 该场景版本不涉及obproxy升级
FAQ： - 升级ob报错：[ERROR] Too many match
– 检查升级命令是否有–usable参数，值为md5值
我是使用admin用户在中控机上安装的obd，我在更新obd的时候，admin用户下提示
[admin@obproxy-node ~]$ obd update
[ERROR] To update OBD, you must be a root user.
我就切换到root用户，然后执行
[root@obproxy-node ~]# /home/admin/oceanbase-all-in-one/obd/usr/bin/obd update
[ERROR] OBD upgrade plugin not found
请问，我该怎么操作呢。
在线升级，把远程镜像打开，离线升级，把升级的包clone到本地镜像目录
您说的这些我都操作了，但还是不能升级。
切换root直接执行obd update ，如果报错 提供下obd的日志
升级 oceanbase操作过程和报错如下：
[admin@obproxy-node ~]$ obd mirror disable remote
Disable remote ok
[admin@obproxy-node ~]$ obd mirror clone /home/admin/soft/oceanbase-all-in-one/rpms/oceanbase-ce-4.0.0.0-102000032022120718.el7.x86_64.rpm
mirror oceanbase-ce-4.0.0.0-102000032022120718.el7.x86_64.rpm existed. Do you want to overwrite? [y/n]: y
name: oceanbase-ce
version: 4.0.0.0
release:102000032022120718.el7
arch: x86_64
md5: e1c71a4cb252e51f210aae129cee0f09613d42bb
add /home/admin/soft/oceanbase-all-in-one/rpms/oceanbase-ce-4.0.0.0-102000032022120718.el7.x86_64.rpm to local mirror
[admin@obproxy-node ~]$
[admin@obproxy-node ~]$ obd mirror list local
±---------------------------------------------------------------------------------------------------------+
| local Package List |
±------------------±--------±-----------------------±-------±-----------------------------------------+
| name | version | release | arch | md5 |
±------------------±--------±-----------------------±-------±-----------------------------------------+
| oceanbase-ce | 4.0.0.0 | 100000282022112511.el7 | x86_64 | 75ec57ebe42718b386901357cef1e5d3864748ee |
| oceanbase-ce-libs | 4.0.0.0 | 100000282022112511.el7 | x86_64 | c5f1e971cc335efe36233cae291a170a0f518e19 |
| ob-deploy | 1.6.2 | 10.el7 | x86_64 | 4542035d1b1af411c93cfe9f1dd78327c2ea05d4 |
| grafana | 7.5.17 | 1 | x86_64 | 1bf1f338d3a3445d8599dc6902e7aeed4de4e0d6 |
| obagent | 1.2.0 | 4.el7 | x86_64 | 0e8f5ee68c337ea28514c9f3f820ea546227fa7e |
| obproxy-ce | 4.0.0 | 5.el7 | x86_64 | de53232a951184fad75b15884458d85e31d2f6c3 |
| oceanbase-ce-libs | 4.0.0.0 | 102000032022120718.el7 | x86_64 | ae630e9c53aeee1f6a542d9bdee6beefdc9af3ad |
| prometheus | 2.37.1 | 10000102022110211.el7 | x86_64 | 58913c7606f05feb01bc1c6410346e5fc31cf263 |
| oceanbase-ce | 4.0.0.0 | 102000032022120718.el7 | x86_64 | e1c71a4cb252e51f210aae129cee0f09613d42bb |
±------------------±--------±-----------------------±-------±-----------------------------------------+
[admin@obproxy-node ~]$ obd cluster upgrade obcluster -c oceanbase-ce -V 4.0.0.0 --usable e1c71a4cb252e51f210aae129cee0f09613d42bb
Get local repositories and plugins ok
±----------------------------------------------------------------------------------------------------+
| Search oceanbase-ce 4.0.0.0 Result |
±-------------±--------±-----------------------±-------±-----------------------------------------+
| name | version | release | arch | md5 |
±-------------±--------±-----------------------±-------±-----------------------------------------+
| oceanbase-ce | 4.0.0.0 | 100000282022112511.el7 | x86_64 | 75ec57ebe42718b386901357cef1e5d3864748ee |
| oceanbase-ce | 4.0.0.0 | 102000032022120718.el7 | x86_64 | e1c71a4cb252e51f210aae129cee0f09613d42bb |
±-------------±--------±-----------------------±-------±-----------------------------------------+
install oceanbase-ce-4.0.0.0 for local ok
Open ssh connection ok
Connect to observer ok
±------------------------------------------------------------------------------------------------------------+
| Packages Will Be Used |
±-------------±--------±-----------------------±-------±-----------------------------------------±------+
| name | version | release | arch | md5 | mark |
±-------------±--------±-----------------------±-------±-----------------------------------------±------+
| oceanbase-ce | 4.0.0.0 | 100000282022112511.el7 | x86_64 | 75ec57ebe42718b386901357cef1e5d3864748ee | start |
| oceanbase-ce | 4.0.0.0 | 102000032022120718.el7 | x86_64 | e1c71a4cb252e51f210aae129cee0f09613d42bb | dest |
±-------------±--------±-----------------------±-------±-----------------------------------------±------+
If you use a non-official release, we cannot guarantee a successful upgrade or technical support when you fail. Make sure that you want to use the above package to upgrade. [y/n]: y
Remote oceanbase-ce-4.0.0.0-102000032022120718.el7-e1c71a4cb252e51f210aae129cee0f09613d42bb repository install ok
Remote oceanbase-ce-4.0.0.0-102000032022120718.el7-e1c71a4cb252e51f210aae129cee0f09613d42bb repository lib check ok
Start observer ok
observer program health check ok
Connect to observer ok
Exec upgrade_checker.py x
See https://www.oceanbase.com/product/ob-deployer/error-codes .
-rw-rw-r-- 1 admin admin 3652 Dec 28 16:49 upgrade_checker.log
[admin@obproxy-node ~]$ more upgrade_checker.log
[2022-12-28 16:49:13] INFO init.py:1611 parameters from cmd: host=“10.110.3.152”, port=2881, user=“root”, password=“ob@Passwd”, log-file=“upgrade_checker.log”
[2022-12-28 16:49:14] INFO init.py:1611 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name=‘min_observer_version’, rowcount = 1
[2022-12-28 16:49:14] INFO init.py:1611 check observer version success, version = 4.0.0.0
[2022-12-28 16:49:14] INFO init.py:1611 succeed to execute query: select count(1) as unsync_cnt from GV$OB_LOG_STAT where in_sync = ‘NO’, rowcount = 1
[2022-12-28 16:49:14] INFO init.py:1611 check paxos replica success
[2022-12-28 16:49:14] INFO init.py:1611 succeed to execute query: select count(1) as cnt from DBA_OB_TENANT_JOBS where job_status=‘INPROGRESS’ and result_code is null, rowcount = 1
[2022-12-28 16:49:14] INFO init.py:1611 succeed to execute query: select count(1) as rebalance_task_cnt from CDB_OB_LS_REPLICA_TASKS, rowcount = 1
[2022-12-28 16:49:14] INFO init.py:1611 check rebalance task success
[2022-12-28 16:49:14] INFO init.py:1611 succeed to execute query: select count(1) from CDB_OB_MAJOR_COMPACTION where STATUS != ‘IDLE’, rowcount = 1
[2022-12-28 16:49:14] INFO init.py:1611 check cluster status success
[2022-12-28 16:49:14] INFO init.py:1611 succeed to execute query: select count(*) as count from DBA_OB_TENANTS where status != ‘NORMAL’, rowcount = 1
[2022-12-28 16:49:14] INFO init.py:1611 check tenant status success
[2022-12-28 16:49:14] INFO init.py:1611 succeed to execute query: select count(1) from CDB_OB_RESTORE_PROGRESS, rowcount = 1
[2022-12-28 16:49:14] INFO init.py:1611 check restore job success
[2022-12-28 16:49:14] INFO init.py:1611 succeed to execute query: select tenant_name,primary_zone from DBA_OB_TENANTS where tenant_id != 1, rowcount = 2
[2022-12-28 16:49:14] INFO init.py:1611 check tenant primary zone success
[2022-12-28 16:49:14] ERROR init.py:1585 run error
Traceback (most recent call last):
File “/tmp/10.110.3.152:2882_10.110.3.153:2882_10.110.3.154:2882/e1c71a4cb252e51f210aae129cee0f09613d42bb/upgrade_checker.py”, line 406, in do_check
check_fail_list()
File “/tmp/10.110.3.152:2882_10.110.3.153:2882_10.110.3.154:2882/e1c71a4cb252e51f210aae129cee0f09613d42bb/upgrade_checker.py”, line 383, in check_fail_list
raise MyError(error_msg)
MyError: 'upgrade checker failed with 2 reasons: [META$1002 tenant primary zone random before update not allowed] , [obuser tenant primary zone random before update not allowed] ’
[2022-12-28 16:49:14] ERROR init.py:1585 normal error
Traceback (most recent call last):
File “/tmp/10.110.3.152:2882_10.110.3.153:2882_10.110.3.154:2882/e1c71a4cb252e51f210aae129cee0f09613d42bb/upgrade_checker.py”, line 410, in do_check
raise e
MyError: 'upgrade checker failed with 2 reasons: [META$1002 tenant primary zone random before update not allowed] , [obuser tenant primary zone random before update not allowed] ’
[2022-12-28 16:49:14] ERROR init.py:1585 normal error
Traceback (most recent call last):
File “/tmp/10.110.3.152:2882_10.110.3.153:2882_10.110.3.154:2882/e1c71a4cb252e51f210aae129cee0f09613d42bb/upgrade_checker.py”, line 440, in
do_check(host, port, user, password, upgrade_params)
File “/tmp/10.110.3.152:2882_10.110.3.153:2882_10.110.3.154:2882/e1c71a4cb252e51f210aae129cee0f09613d42bb/upgrade_checker.py”, line 419, in do_check
raise e
MyError: 'upgrade checker failed with 2 reasons: [META$1002 tenant primary zone random before update not allowed] , [obuser tenant primary zone random before update not allowed] ’
[admin@obproxy-node .obd]$ obd cluster list
±-----------------------------------------------------------------+
| Cluster List |
±----------±-----------------------------------±----------------+
| Name | Configuration Path | Status (Cached) |
±----------±-----------------------------------±----------------+
| demo | /home/admin/.obd/cluster/demo | configured |
| obcluster | /home/admin/.obd/cluster/obcluster | upgrading |
±----------±-----------------------------------±----------------+
报错挺清晰的，升级的时候，需要设置primary zone，这里没有设置primary zone升级的检测就直接报错
1002租户指定下主zone，ALTER TENANT tenant1 primary_zone=‘zone1’;
请问当前这种情况，我该如何继续升级呢。
您好，现在数据库中途升级不成功，就昨天如下这种状态：
[admin@obproxy-node .obd]$ obd cluster list
±-----------------------------------------------------------------+
| Cluster List |
±----------±-----------------------------------±----------------+
| Name | Configuration Path | Status (Cached) |
±----------±-----------------------------------±----------------+
| demo | /home/admin/.obd/cluster/demo | configured |
| obcluster | /home/admin/.obd/cluster/obcluster | upgrading |
±----------±-----------------------------------±----------------+
我接下来该怎么继续操作升级呢。
[admin@obproxy-node ~]$ obclient -h 10.110.3.152 -uroot@sys#obcluster -pob@Passwd -P2883 -c -A oceanbase
Welcome to the OceanBase. Commands end with ; or \g.
Your OceanBase connection id is 42
Server version: OceanBase_CE 4.0.0.0 (r100000282022112511-dd289d2407609a88b1fcdf2be9e7c384cb8e19d0) (Built Nov 25 2022 11:58:08)
Copyright (c) 2000, 2018, OceanBase and/or its affiliates. All rights reserved.
Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.
obclient [oceanbase]> ALTER TENANT tenant1 primary_zone=‘zone1’;
ERROR 5157 (HY000): Unknown tenant ‘tenant1’
这样执行报错，请问该如何执行。
我创建的多租户是 obuser ，我将obuser 的PRIMARY_ZONE 修改为 ‘zone1’;
[admin@obproxy-node conf]$ obclient -h 10.110.3.152 -uroot@sys#obcluster -pob@Passwd -P2883 -c -A oceanbase
Welcome to the OceanBase. Commands end with ; or \g.
Your OceanBase connection id is 46
Server version: OceanBase_CE 4.0.0.0 (r100000282022112511-dd289d2407609a88b1fcdf2be9e7c384cb8e19d0) (Built Nov 25 2022 11:58:08)
Copyright (c) 2000, 2018, OceanBase and/or its affiliates. All rights reserved.
Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.
obclient [oceanbase]> ALTER TENANT obuser PRIMARY_ZONE =‘zone1’;
Query OK, 0 rows affected (0.835 sec)
[admin@obproxy-node ~]$ obclient -h 10.110.3.152 -uroot@sys#obcluster -pob@Passwd -P2883 -c -A oceanbase
Welcome to the OceanBase. Commands end with ; or \g.
Your OceanBase connection id is 50
Server version: OceanBase_CE 4.0.0.0 (r102000032022120718-58fdb0ef1a9b589ef05e56c1ede65e951986d996) (Built Dec 7 2022 18:32:31)
Copyright (c) 2000, 2018, OceanBase and/or its affiliates. All rights reserved.
Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.
obclient [oceanbase]> ALTER TENANT obuser PRIMARY_ZONE =‘zone2’;
Query OK, 0 rows affected (0.249 sec)
obclient [oceanbase]> ALTER TENANT obuser PRIMARY_ZONE =‘zone1;zone2;zone3’;
Query OK, 0 rows affected (0.095 sec)
再执行：obd cluster upgrade obcluster -c oceanbase-ce -V 4.0.0.0 --usable e1c71a4cb252e51f210aae129cee0f09613d42bb 升级成功
[admin@obproxy-node ~]$ obd cluster upgrade obcluster -c oceanbase-ce -V 4.0.0.0 --usable e1c71a4cb252e51f210aae129cee0f09613d42bb
Get local repositories and plugins ok
Open ssh connection ok
Remote oceanbase-ce-4.0.0.0-102000032022120718.el7-e1c71a4cb252e51f210aae129cee0f09613d42bb repository install ok
Remote oceanbase-ce-4.0.0.0-102000032022120718.el7-e1c71a4cb252e51f210aae129cee0f09613d42bb repository lib check ok
Start observer ok
observer program health check ok
Connect to observer ok
Exec upgrade_checker.py ok
Enable upgrade mode ok
Exec upgrade_pre.py ok
Rotation upgrade ok
upgrade zone “zone1”
Remote oceanbase-ce-4.0.0.0-102000032022120718.el7-e1c71a4cb252e51f210aae129cee0f09613d42bb repository lib check ok
Stop observer ok
Start observer ok
observer program health check ok
Connect to observer ok
upgrade zone “zone2”
Remote oceanbase-ce-4.0.0.0-102000032022120718.el7-e1c71a4cb252e51f210aae129cee0f09613d42bb repository install ok
Remote oceanbase-ce-4.0.0.0-102000032022120718.el7-e1c71a4cb252e51f210aae129cee0f09613d42bb repository lib check ok
Stop observer ok
Start observer ok
observer program health check ok
Connect to observer ok
upgrade zone “zone3”
Remote oceanbase-ce-4.0.0.0-102000032022120718.el7-e1c71a4cb252e51f210aae129cee0f09613d42bb repository install ok
Remote oceanbase-ce-4.0.0.0-102000032022120718.el7-e1c71a4cb252e51f210aae129cee0f09613d42bb repository lib check ok
Stop observer ok
Start observer ok
observer program health check ok
Connect to observer ok
Exec upgrade_post.py ok
Disable upgrade mode ok
Root inspection ok
Exec upgrade_post_checker.py ok
Wait for observer init ok
±-----------------------------------------------+
| observer |
±-------------±--------±-----±------±-------+
| ip | version | port | zone | status |
±-------------±--------±-----±------±-------+
| 10.110.3.152 | 4.0.0.0 | 2881 | zone1 | ACTIVE |
| 10.110.3.153 | 4.0.0.0 | 2881 | zone2 | ACTIVE |
| 10.110.3.154 | 4.0.0.0 | 2881 | zone3 | ACTIVE |
±-------------±--------±-----±------±-------+
obclient -h10.110.3.152 -P2881 -uroot -pob@Passwd -Doceanbase -A