【 使用环境 】 测试环境
【 OB or 其他组件 】observer
【 使用版本 】4.0.0
【问题描述】ob4.0在线升级ob4.1报错
现在集群状态为
集群无法启动，如何处理？
[2023-06-12 17:40:04.484] [DEBUG] - cmd: [‘dllg_cluster’]
[2023-06-12 17:40:04.485] [DEBUG] - opts: {‘component’: ‘oceanbase-ce’, ‘version’: ‘4.1.0.0’, ‘skip_check’: None, ‘usable’: ‘66882706a45d94c379dce7fd6c0b3c9c33cd780e95658d3a475029013a4b3451’, ‘disable’: ‘’, ‘executer_path’: ‘/root/.oceanbase-all-in-one/obd/usr/obd/lib/executer’, ‘script_query_timeout’: ‘’}
[2023-06-12 17:40:04.485] [DEBUG] - mkdir /root/.obd/lock/
[2023-06-12 17:40:04.485] [DEBUG] - unknown lock mode
[2023-06-12 17:40:04.486] [DEBUG] - try to get share lock /root/.obd/lock/global
[2023-06-12 17:40:04.486] [DEBUG] - share lock /root/.obd/lock/global, count 1
[2023-06-12 17:40:04.486] [DEBUG] - Get Deploy by name
[2023-06-12 17:40:04.486] [DEBUG] - mkdir /root/.obd/cluster/
[2023-06-12 17:40:04.486] [DEBUG] - mkdir /root/.obd/config_parser/
[2023-06-12 17:40:04.487] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_dllg_cluster
[2023-06-12 17:40:04.487] [DEBUG] - exclusive lock /root/.obd/lock/deploy_dllg_cluster, count 1
[2023-06-12 17:40:04.495] [DEBUG] - Deploy status judge
[2023-06-12 17:40:04.559] [INFO] Get local repositories and plugins
[2023-06-12 17:40:04.559] [DEBUG] - mkdir /root/.obd/repository
[2023-06-12 17:40:04.560] [DEBUG] - Get local repository oceanbase-ce-4.0.0.0-1d56dc742f5f05a2d15797d291b51a94019e728d
[2023-06-12 17:40:04.561] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo
[2023-06-12 17:40:04.561] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 1
[2023-06-12 17:40:04.564] [DEBUG] - Get local repository obproxy-ce-4.0.0-de53232a951184fad75b15884458d85e31d2f6c3
[2023-06-12 17:40:04.564] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 2
[2023-06-12 17:40:04.567] [DEBUG] - Searching param plugin for components …
[2023-06-12 17:40:04.568] [DEBUG] - Search param plugin for oceanbase-ce
[2023-06-12 17:40:04.568] [DEBUG] - mkdir /root/.obd/plugins
[2023-06-12 17:40:04.569] [DEBUG] - Found for oceanbase-ce-param-4.0.0.0 for oceanbase-ce-4.0.0.0
[2023-06-12 17:40:04.569] [DEBUG] - Applying oceanbase-ce-param-4.0.0.0 for oceanbase-ce-4.0.0.0-103000022023011215.el7-1d56dc742f5f05a2d15797d291b51a94019e728d
[2023-06-12 17:40:05.439] [DEBUG] - Search param plugin for obproxy-ce
[2023-06-12 17:40:05.439] [DEBUG] - Found for obproxy-ce-param-3.1.0 for obproxy-ce-4.0.0
[2023-06-12 17:40:05.439] [DEBUG] - Applying obproxy-ce-param-3.1.0 for obproxy-ce-4.0.0-5.el7-de53232a951184fad75b15884458d85e31d2f6c3
[2023-06-12 17:40:05.757] [DEBUG] - Search repository oceanbase-ce version: 4.0.0.0, tag: 1d56dc742f5f05a2d15797d291b51a94019e728d, release: None, package_hash: None
[2023-06-12 17:40:05.758] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 3
[2023-06-12 17:40:05.758] [DEBUG] - mkdir /root/.obd/repository/oceanbase-ce
[2023-06-12 17:40:05.758] [DEBUG] - Found repository oceanbase-ce-4.0.0.0-103000022023011215.el7-1d56dc742f5f05a2d15797d291b51a94019e728d
[2023-06-12 17:40:05.758] [DEBUG] - Search repository oceanbase-ce version: 4.1.0.0, tag: 21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe, release: None, package_hash: None
[2023-06-12 17:40:05.760] [DEBUG] - Found repository oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe
[2023-06-12 17:40:05.761] [INFO] Open ssh connection
[2023-06-12 17:40:06.544] [DEBUG] - Searching install plugin for components …
[2023-06-12 17:40:06.545] [DEBUG] - Search install plugin for oceanbase-ce
[2023-06-12 17:40:06.545] [DEBUG] - Found for oceanbase-ce-install-4.0.0.0 for oceanbase-ce-4.0.0.0
[2023-06-12 17:40:06.545] [DEBUG] - Search install plugin for oceanbase-ce
[2023-06-12 17:40:06.545] [DEBUG] - Found for oceanbase-ce-install-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-06-12 17:40:06.545] [DEBUG] - Searching install plugin for components …
[2023-06-12 17:40:06.551] [INFO] Open ssh connection
[2023-06-12 17:40:07.204] [DEBUG] - Call general-py_script_install_repo-0.1 for oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe
[2023-06-12 17:40:07.204] [DEBUG] - import install_repo
[2023-06-12 17:40:07.208] [DEBUG] - add install_repo ref count to 1
[2023-06-12 17:40:07.209] [INFO] Remote oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe repository install
[2023-06-12 17:40:07.210] [DEBUG] – Remote oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe repository integrity check
[2023-06-12 17:40:07.210] [DEBUG] – server1(192.168.0.192) oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe repository integrity check
[2023-06-12 17:40:07.210] [DEBUG] – admin@192.168.0.192 execute: echo ${OBD_HOME:-"$HOME"}/.obd
[2023-06-12 17:40:07.664] [DEBUG] – exited code 0
[2023-06-12 17:40:07.664] [DEBUG] – admin@192.168.0.192 execute: cat /home/admin/.obd/repository/oceanbase-ce/4.1.0.0/21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe/.data
[2023-06-12 17:40:08.137] [DEBUG] – exited code 0
[2023-06-12 17:40:08.137] [DEBUG] – server1(192.168.0.192) oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe install check
[2023-06-12 17:40:08.141] [DEBUG] – server1(192.168.0.192) oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe has installed
[2023-06-12 17:40:08.142] [DEBUG] – admin@192.168.0.192 set env _repo_dir to ‘/home/admin/.obd/repository/oceanbase-ce/4.1.0.0/21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe’
[2023-06-12 17:40:08.142] [DEBUG] – admin@192.168.0.192 set env _home_path to ‘/app/oceanbase/home/observer’
[2023-06-12 17:40:08.142] [DEBUG] – admin@192.168.0.192 execute: mkdir -p ${_home_path} && cd ${_repo_dir} && find -type d | xargs -i mkdir -p ${_home_path}/{}
[2023-06-12 17:40:08.725] [DEBUG] – exited code 0
[2023-06-12 17:40:08.726] [DEBUG] – admin@192.168.0.192 set env source to ‘/home/admin/.obd/repository/oceanbase-ce/4.1.0.0/21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe/bin/observer’
[2023-06-12 17:40:08.726] [DEBUG] – admin@192.168.0.192 set env target to ‘/app/oceanbase/home/observer/bin/observer’
[2023-06-12 17:40:08.726] [DEBUG] – admin@192.168.0.192 execute: ln -fs ${source} ${target}
[2023-06-12 17:40:09.458] [DEBUG] – exited code 0
[2023-06-12 17:40:09.459] [DEBUG] – admin@192.168.0.192 set env source to ‘/home/admin/.obd/repository/oceanbase-ce/4.1.0.0/21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe/bin’
[2023-06-12 17:40:09.459] [DEBUG] – admin@192.168.0.192 set env target to ‘/app/oceanbase/home/observer/bin’
[2023-06-12 17:40:09.460] [DEBUG] – admin@192.168.0.192 execute: ls -1 ${source}
[2023-06-12 17:40:09.721] [DEBUG] – exited code 0
[2023-06-12 17:40:09.722] [DEBUG] – admin@192.168.0.192 execute: cd ${source} && find -type f | xargs -i ln -fs ${source}/{} ${target}/{}
[2023-06-12 17:40:11.417] [DEBUG] – exited code 0
[2023-06-12 17:40:11.418] [DEBUG] – admin@192.168.0.192 execute: cd ${source} && find -type l | xargs -i ln -fs ${source}/{} ${target}/{}
[2023-06-12 17:40:12.141] [DEBUG] – exited code 0
[2023-06-12 17:40:12.141] [DEBUG] – admin@192.168.0.192 set env source to ‘/home/admin/.obd/repository/oceanbase-ce/4.1.0.0/21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe/etc’
[2023-06-12 17:40:12.142] [DEBUG] – admin@192.168.0.192 set env target to ‘/app/oceanbase/home/observer/etc’
[2023-06-12 17:40:12.142] [DEBUG] – admin@192.168.0.192 execute: ls -1 ${source}
[2023-06-12 17:40:12.530] [DEBUG] – exited code 0
[2023-06-12 17:40:12.530] [DEBUG] – admin@192.168.0.192 execute: cd ${source} && find -type f | xargs -i cp -f ${source}/{} ${target}/{}
[2023-06-12 17:40:15.490] [DEBUG] – exited code 0
[2023-06-12 17:40:15.491] [DEBUG] – admin@192.168.0.192 execute: cd ${source} && find -type l | xargs -i cp -f ${source}/{} ${target}/{}
[2023-06-12 17:40:15.850] [DEBUG] – exited code 0
[2023-06-12 17:40:15.851] [DEBUG] – admin@192.168.0.192 set env source to ‘/home/admin/.obd/repository/oceanbase-ce/4.1.0.0/21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe/admin’
[2023-06-12 17:40:15.851] [DEBUG] – admin@192.168.0.192 set env target to ‘/app/oceanbase/home/observer/admin’
[2023-06-12 17:40:15.851] [DEBUG] – admin@192.168.0.192 execute: ls -1 ${source}
[2023-06-12 17:40:16.383] [DEBUG] – exited code 0
[2023-06-12 17:40:16.384] [DEBUG] – admin@192.168.0.192 execute: cd ${source} && find -type f | xargs -i ln -fs ${source}/{} ${target}/{}
[2023-06-12 17:40:46.494] [ERROR] Traceback (most recent call last):
[2023-06-12 17:40:46.494] [ERROR] File “paramiko/channel.py”, line 699, in recv
[2023-06-12 17:40:46.494] [ERROR] File “paramiko/buffered_pipe.py”, line 164, in read
[2023-06-12 17:40:46.494] [ERROR] paramiko.buffered_pipe.PipeTimeout
[2023-06-12 17:40:46.494] [ERROR]
[2023-06-12 17:40:46.494] [ERROR] During handling of the above exception, another exception occurred:
[2023-06-12 17:40:46.494] [ERROR]
[2023-06-12 17:40:46.494] [ERROR] Traceback (most recent call last):
[2023-06-12 17:40:46.494] [ERROR] File “core.py”, line 2745, in upgrade_cluster
[2023-06-12 17:40:46.495] [ERROR] File “core.py”, line 1510, in install_repositories_to_servers
[2023-06-12 17:40:46.495] [ERROR] File “core.py”, line 184, in call_plugin
[2023-06-12 17:40:46.495] [ERROR] File “_plugin.py”, line 323, in call
[2023-06-12 17:40:46.495] [ERROR] File “_plugin.py”, line 285, in _new_func
[2023-06-12 17:40:46.495] [ERROR] File “/root/.obd/plugins/general/0.1/install_repo.py”, line 105, in install_repo
[2023-06-12 17:40:46.495] [ERROR] if not install_to_home_path():
[2023-06-12 17:40:46.495] [ERROR] File “/root/.obd/plugins/general/0.1/install_repo.py”, line 58, in install_to_home_path
[2023-06-12 17:40:46.495] [ERROR] success = client.execute_command(“cd ${source} && find -type f | xargs -i %(install_cmd)s ${source}/{} ${target}/{}” % {“install_cmd”: install_cmd}) and success
[2023-06-12 17:40:46.495] [ERROR] File “_plugin.py”, line 222, in new_method
[2023-06-12 17:40:46.495] [ERROR] File “_stdio.py”, line 885, in func_wrapper
[2023-06-12 17:40:46.495] [ERROR] File “ssh.py”, line 450, in execute_command
[2023-06-12 17:40:46.495] [ERROR] File “_stdio.py”, line 885, in func_wrapper
[2023-06-12 17:40:46.495] [ERROR] File “ssh.py”, line 412, in _execute_command
[2023-06-12 17:40:46.495] [ERROR] File “paramiko/file.py”, line 200, in read
[2023-06-12 17:40:46.495] [ERROR] File “paramiko/channel.py”, line 1361, in _read
[2023-06-12 17:40:46.495] [ERROR] File “paramiko/channel.py”, line 701, in recv
[2023-06-12 17:40:46.495] [ERROR] socket.timeout
[2023-06-12 17:40:46.495] [ERROR]
[2023-06-12 17:40:46.496] [DEBUG] – admin@192.168.0.192 execute: cd ${source} && find -type l | xargs -i ln -fs ${source}/{} ${target}/{}
[2023-06-12 17:40:46.945] [DEBUG] – exited code 0
[2023-06-12 17:40:46.945] [ERROR] Failed to install repository oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe to /app/oceanbase/home/observer
[2023-06-12 17:40:46.946] [DEBUG] - sub install_repo ref count to 0
[2023-06-12 17:40:46.946] [DEBUG] - export install_repo
[2023-06-12 17:40:46.949] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2023-06-12 17:40:46.950] [INFO] Trace ID: 1c080f1a-0905-11ee-99f1-fa163e4bac35
[2023-06-12 17:40:46.950] [INFO] If you want to view detailed obd logs, please run: obd display-trace 1c080f1a-0905-11ee-99f1-fa163e4bac35
[2023-06-12 17:40:46.950] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 2
[2023-06-12 17:40:46.950] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1
[2023-06-12 17:40:46.950] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[2023-06-12 17:40:46.950] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[2023-06-12 17:40:46.950] [DEBUG] - exclusive lock /root/.obd/lock/deploy_dllg_cluster release, count 0
[2023-06-12 17:40:46.950] [DEBUG] - unlock /root/.obd/lock/deploy_dllg_cluster
[2023-06-12 17:40:46.951] [DEBUG] - share lock /root/.obd/lock/global release, count 0
[2023-06-12 17:40:46.951] [DEBUG] - unlock /root/.obd/lock/global
[2023-06-12 17:40:46.973] [INFO] [ERROR] Failed to install repository oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe to /app/oceanbase/home/observer
[2023-06-12 17:40:46.973] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2023-06-12 17:40:46.973] [INFO] Trace ID: 1c080f1a-0905-11ee-99f1-fa163e4bac35
[2023-06-12 17:40:46.973] [INFO] If you want to view detailed obd logs, please run: obd display-trace 1c080f1a-0905-11ee-99f1-fa163e4bac35
[2023-06-12 17:40:46.973] [INFO]
1 个赞
看着像是ssh不通，连接超时了，检查一下ssh？？？
2023-06-12 17:40:46.495] [ERROR] File “/root/.obd/plugins/general/0.1/install_repo.py”, line 58, in install_to_home_path
[2023-06-12 17:40:46.495] [ERROR] success = client.execute_command(“cd ${source} && find -type f | xargs -i %(install_cmd)s ${source}/{} ${target}/{}” % {“install_cmd”: install_cmd}) and success
[2023-06-12 17:40:46.495] [ERROR] File “_plugin.py”, line 222, in new_method
[2023-06-12 17:40:46.495] [ERROR] File “_stdio.py”, line 885, in func_wrapper
[2023-06-12 17:40:46.495] [ERROR] File “ssh.py”, line 450, in execute_command
[2023-06-12 17:40:46.495] [ERROR] File “_stdio.py”, line 885, in func_wrapper
[2023-06-12 17:40:46.495] [ERROR] File “ssh.py”, line 412, in _execute_command
[2023-06-12 17:40:46.495] [ERROR] File “paramiko/file.py”, line 200, in read
[2023-06-12 17:40:46.495] [ERROR] File “paramiko/channel.py”, line 1361, in _read
[2023-06-12 17:40:46.495] [ERROR] File “paramiko/channel.py”, line 701, in recv
[2023-06-12 17:40:46.495] [ERROR] socket.timeout
1 个赞
ssh是通的，原来集群是正常运行的，可以停止和重启，只是升级过程中执行DDL出错，就升级终止，停留在升级状态，能否回退或完成升级？
先手动ssh 一下中控机到各个节点，确定连通性没有问题之后 可以重新执行你昨天执行的升级命令。obd支持升级中断后 继续升级（内部有记录断点位置的）
ob目前应该是不支持回退版本。
就再次尝试发起升级试试吧。如果检查都是通的，那可能只是升级升级耗时较长，超时中断了。再次发起试试，如果还错，继续看错误日志，看看是不是又向下做了一部分，多发起几次说不定就能完成。
还是不行.
[root@control ~]# obd display-trace 86e80ca2-09ac-11ee-b959-fa163e4bac35
[2023-06-13 13:38:29.744] [DEBUG] - cmd: [‘dllg_cluster’]
[2023-06-13 13:38:29.745] [DEBUG] - opts: {‘component’: ‘oceanbase-ce’, ‘version’: ‘4.1.0.0’, ‘skip_check’: None, ‘usable’: ‘66882706a45d94c379dce7fd6c0b3c9c33cd780e95658d3a475029013a4b3451’, ‘disable’: ‘’, ‘executer_path’: ‘/root/.oceanbase-all-in-one/obd/usr/obd/lib/executer’, ‘script_query_timeout’: ‘’}
[2023-06-13 13:38:29.745] [DEBUG] - mkdir /root/.obd/lock/
[2023-06-13 13:38:29.745] [DEBUG] - unknown lock mode
[2023-06-13 13:38:29.745] [DEBUG] - try to get share lock /root/.obd/lock/global
[2023-06-13 13:38:29.745] [DEBUG] - share lock /root/.obd/lock/global, count 1
[2023-06-13 13:38:29.745] [DEBUG] - Get Deploy by name
[2023-06-13 13:38:29.745] [DEBUG] - mkdir /root/.obd/cluster/
[2023-06-13 13:38:29.746] [DEBUG] - mkdir /root/.obd/config_parser/
[2023-06-13 13:38:29.746] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_dllg_cluster
[2023-06-13 13:38:29.746] [DEBUG] - exclusive lock /root/.obd/lock/deploy_dllg_cluster, count 1
[2023-06-13 13:38:29.751] [DEBUG] - Deploy status judge
[2023-06-13 13:38:29.787] [INFO] Get local repositories and plugins
[2023-06-13 13:38:29.788] [DEBUG] - mkdir /root/.obd/repository
[2023-06-13 13:38:29.788] [DEBUG] - Get local repository oceanbase-ce-4.0.0.0-1d56dc742f5f05a2d15797d291b51a94019e728d
[2023-06-13 13:38:29.789] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo
[2023-06-13 13:38:29.789] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 1
[2023-06-13 13:38:29.791] [DEBUG] - Get local repository obproxy-ce-4.0.0-de53232a951184fad75b15884458d85e31d2f6c3
[2023-06-13 13:38:29.791] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 2
[2023-06-13 13:38:29.793] [DEBUG] - Searching param plugin for components …
[2023-06-13 13:38:29.793] [DEBUG] - Search param plugin for oceanbase-ce
[2023-06-13 13:38:29.793] [DEBUG] - mkdir /root/.obd/plugins
[2023-06-13 13:38:29.794] [DEBUG] - Found for oceanbase-ce-param-4.0.0.0 for oceanbase-ce-4.0.0.0
[2023-06-13 13:38:29.794] [DEBUG] - Applying oceanbase-ce-param-4.0.0.0 for oceanbase-ce-4.0.0.0-103000022023011215.el7-1d56dc742f5f05a2d15797d291b51a94019e728d
[2023-06-13 13:38:30.320] [DEBUG] - Search param plugin for obproxy-ce
[2023-06-13 13:38:30.321] [DEBUG] - Found for obproxy-ce-param-3.1.0 for obproxy-ce-4.0.0
[2023-06-13 13:38:30.321] [DEBUG] - Applying obproxy-ce-param-3.1.0 for obproxy-ce-4.0.0-5.el7-de53232a951184fad75b15884458d85e31d2f6c3
[2023-06-13 13:38:30.560] [DEBUG] - Search repository oceanbase-ce version: 4.0.0.0, tag: 1d56dc742f5f05a2d15797d291b51a94019e728d, release: None, package_hash: None
[2023-06-13 13:38:30.560] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 3
[2023-06-13 13:38:30.560] [DEBUG] - mkdir /root/.obd/repository/oceanbase-ce
[2023-06-13 13:38:30.560] [DEBUG] - Found repository oceanbase-ce-4.0.0.0-103000022023011215.el7-1d56dc742f5f05a2d15797d291b51a94019e728d
[2023-06-13 13:38:30.561] [DEBUG] - Search repository oceanbase-ce version: 4.1.0.0, tag: 21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe, release: None, package_hash: None
[2023-06-13 13:38:30.564] [DEBUG] - Found repository oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe
[2023-06-13 13:38:30.565] [INFO] Open ssh connection
[2023-06-13 13:38:31.870] [DEBUG] - Searching install plugin for components …
[2023-06-13 13:38:31.870] [DEBUG] - Search install plugin for oceanbase-ce
[2023-06-13 13:38:31.871] [DEBUG] - Found for oceanbase-ce-install-4.0.0.0 for oceanbase-ce-4.0.0.0
[2023-06-13 13:38:31.871] [DEBUG] - Search install plugin for oceanbase-ce
[2023-06-13 13:38:31.871] [DEBUG] - Found for oceanbase-ce-install-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-06-13 13:38:31.871] [DEBUG] - Searching install plugin for components …
[2023-06-13 13:38:31.880] [INFO] Open ssh connection
[2023-06-13 13:38:32.664] [DEBUG] - Call general-py_script_install_repo-0.1 for oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe
[2023-06-13 13:38:32.664] [DEBUG] - import install_repo
[2023-06-13 13:38:32.666] [DEBUG] - add install_repo ref count to 1
[2023-06-13 13:38:32.667] [INFO] Remote oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe repository install
[2023-06-13 13:38:32.668] [DEBUG] – Remote oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe repository integrity check
[2023-06-13 13:38:32.668] [DEBUG] – server1(192.168.0.192) oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe repository integrity check
[2023-06-13 13:38:32.668] [DEBUG] – admin@192.168.0.192 execute: echo ${OBD_HOME:-"$HOME"}/.obd
[2023-06-13 13:38:33.092] [DEBUG] – exited code 0
[2023-06-13 13:38:33.092] [DEBUG] – admin@192.168.0.192 execute: cat /home/admin/.obd/repository/oceanbase-ce/4.1.0.0/21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe/.data
[2023-06-13 13:38:33.520] [DEBUG] – exited code 0
[2023-06-13 13:38:33.521] [DEBUG] – server1(192.168.0.192) oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe install check
[2023-06-13 13:38:33.524] [DEBUG] – server1(192.168.0.192) oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe has installed
[2023-06-13 13:38:33.525] [DEBUG] – admin@192.168.0.192 set env _repo_dir to ‘/home/admin/.obd/repository/oceanbase-ce/4.1.0.0/21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe’
[2023-06-13 13:38:33.525] [DEBUG] – admin@192.168.0.192 set env _home_path to ‘/app/oceanbase/home/observer’
[2023-06-13 13:38:33.525] [DEBUG] – admin@192.168.0.192 execute: mkdir -p ${_home_path} && cd ${_repo_dir} && find -type d | xargs -i mkdir -p ${_home_path}/{}
[2023-06-13 13:38:35.480] [DEBUG] – exited code 0
[2023-06-13 13:38:35.480] [DEBUG] – admin@192.168.0.192 set env source to ‘/home/admin/.obd/repository/oceanbase-ce/4.1.0.0/21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe/bin/observer’
[2023-06-13 13:38:35.481] [DEBUG] – admin@192.168.0.192 set env target to ‘/app/oceanbase/home/observer/bin/observer’
[2023-06-13 13:38:35.481] [DEBUG] – admin@192.168.0.192 execute: ln -fs ${source} ${target}
[2023-06-13 13:38:37.465] [DEBUG] – exited code 0
[2023-06-13 13:38:37.466] [DEBUG] – admin@192.168.0.192 set env source to ‘/home/admin/.obd/repository/oceanbase-ce/4.1.0.0/21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe/bin’
[2023-06-13 13:38:37.467] [DEBUG] – admin@192.168.0.192 set env target to ‘/app/oceanbase/home/observer/bin’
[2023-06-13 13:38:37.467] [DEBUG] – admin@192.168.0.192 execute: ls -1 ${source}
[2023-06-13 13:38:38.028] [DEBUG] – exited code 0
[2023-06-13 13:38:38.029] [DEBUG] – admin@192.168.0.192 execute: cd ${source} && find -type f | xargs -i ln -fs ${source}/{} ${target}/{}
[2023-06-13 13:38:39.457] [DEBUG] – exited code 0
[2023-06-13 13:38:39.458] [DEBUG] – admin@192.168.0.192 execute: cd ${source} && find -type l | xargs -i ln -fs ${source}/{} ${target}/{}
[2023-06-13 13:38:39.834] [DEBUG] – exited code 0
[2023-06-13 13:38:39.834] [DEBUG] – admin@192.168.0.192 set env source to ‘/home/admin/.obd/repository/oceanbase-ce/4.1.0.0/21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe/etc’
[2023-06-13 13:38:39.835] [DEBUG] – admin@192.168.0.192 set env target to ‘/app/oceanbase/home/observer/etc’
[2023-06-13 13:38:39.835] [DEBUG] – admin@192.168.0.192 execute: ls -1 ${source}
[2023-06-13 13:38:40.385] [DEBUG] – exited code 0
[2023-06-13 13:38:40.386] [DEBUG] – admin@192.168.0.192 execute: cd ${source} && find -type f | xargs -i cp -f ${source}/{} ${target}/{}
[2023-06-13 13:38:46.446] [DEBUG] – exited code 0
[2023-06-13 13:38:46.447] [DEBUG] – admin@192.168.0.192 execute: cd ${source} && find -type l | xargs -i cp -f ${source}/{} ${target}/{}
[2023-06-13 13:38:46.947] [DEBUG] – exited code 0
[2023-06-13 13:38:46.947] [DEBUG] – admin@192.168.0.192 set env source to ‘/home/admin/.obd/repository/oceanbase-ce/4.1.0.0/21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe/admin’
[2023-06-13 13:38:46.948] [DEBUG] – admin@192.168.0.192 set env target to ‘/app/oceanbase/home/observer/admin’
[2023-06-13 13:38:46.948] [DEBUG] – admin@192.168.0.192 execute: ls -1 ${source}
[2023-06-13 13:38:47.410] [DEBUG] – exited code 0
[2023-06-13 13:38:47.410] [DEBUG] – admin@192.168.0.192 execute: cd ${source} && find -type f | xargs -i ln -fs ${source}/{} ${target}/{}
[2023-06-13 13:39:17.476] [ERROR] Traceback (most recent call last):
[2023-06-13 13:39:17.476] [ERROR] File “paramiko/channel.py”, line 699, in recv
[2023-06-13 13:39:17.477] [ERROR] File “paramiko/buffered_pipe.py”, line 164, in read
[2023-06-13 13:39:17.477] [ERROR] paramiko.buffered_pipe.PipeTimeout
[2023-06-13 13:39:17.477] [ERROR]
[2023-06-13 13:39:17.477] [ERROR] During handling of the above exception, another exception occurred:
[2023-06-13 13:39:17.477] [ERROR]
[2023-06-13 13:39:17.477] [ERROR] Traceback (most recent call last):
[2023-06-13 13:39:17.477] [ERROR] File “core.py”, line 2745, in upgrade_cluster
[2023-06-13 13:39:17.477] [ERROR] File “core.py”, line 1510, in install_repositories_to_servers
[2023-06-13 13:39:17.477] [ERROR] File “core.py”, line 184, in call_plugin
[2023-06-13 13:39:17.477] [ERROR] File “_plugin.py”, line 323, in call
[2023-06-13 13:39:17.477] [ERROR] File “_plugin.py”, line 285, in _new_func
[2023-06-13 13:39:17.477] [ERROR] File “/root/.obd/plugins/general/0.1/install_repo.py”, line 105, in install_repo
[2023-06-13 13:39:17.477] [ERROR] if not install_to_home_path():
[2023-06-13 13:39:17.477] [ERROR] File “/root/.obd/plugins/general/0.1/install_repo.py”, line 58, in install_to_home_path
[2023-06-13 13:39:17.477] [ERROR] success = client.execute_command(“cd ${source} && find -type f | xargs -i %(install_cmd)s ${source}/{} ${target}/{}” % {“install_cmd”: install_cmd}) and success
[2023-06-13 13:39:17.477] [ERROR] File “_plugin.py”, line 222, in new_method
[2023-06-13 13:39:17.477] [ERROR] File “_stdio.py”, line 885, in func_wrapper
[2023-06-13 13:39:17.477] [ERROR] File “ssh.py”, line 450, in execute_command
[2023-06-13 13:39:17.477] [ERROR] File “_stdio.py”, line 885, in func_wrapper
[2023-06-13 13:39:17.477] [ERROR] File “ssh.py”, line 412, in _execute_command
[2023-06-13 13:39:17.478] [ERROR] File “paramiko/file.py”, line 200, in read
[2023-06-13 13:39:17.478] [ERROR] File “paramiko/channel.py”, line 1361, in _read
[2023-06-13 13:39:17.478] [ERROR] File “paramiko/channel.py”, line 701, in recv
[2023-06-13 13:39:17.478] [ERROR] socket.timeout
[2023-06-13 13:39:17.478] [ERROR]
[2023-06-13 13:39:17.478] [DEBUG] – admin@192.168.0.192 execute: cd ${source} && find -type l | xargs -i ln -fs ${source}/{} ${target}/{}
[2023-06-13 13:39:17.824] [DEBUG] – exited code 0
[2023-06-13 13:39:17.825] [ERROR] Failed to install repository oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe to /app/oceanbase/home/observer
[2023-06-13 13:39:17.825] [DEBUG] - sub install_repo ref count to 0
[2023-06-13 13:39:17.825] [DEBUG] - export install_repo
[2023-06-13 13:39:17.830] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2023-06-13 13:39:17.830] [INFO] Trace ID: 86e80ca2-09ac-11ee-b959-fa163e4bac35
[2023-06-13 13:39:17.830] [INFO] If you want to view detailed obd logs, please run: obd display-trace 86e80ca2-09ac-11ee-b959-fa163e4bac35
[2023-06-13 13:39:17.831] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 2
[2023-06-13 13:39:17.831] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1
[2023-06-13 13:39:17.831] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[2023-06-13 13:39:17.831] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[2023-06-13 13:39:17.831] [DEBUG] - exclusive lock /root/.obd/lock/deploy_dllg_cluster release, count 0
[2023-06-13 13:39:17.831] [DEBUG] - unlock /root/.obd/lock/deploy_dllg_cluster
[2023-06-13 13:39:17.831] [DEBUG] - share lock /root/.obd/lock/global release, count 0
[2023-06-13 13:39:17.831] [DEBUG] - unlock /root/.obd/lock/global
[2023-06-13 13:39:17.905] [INFO] [ERROR] Failed to install repository oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe to /app/oceanbase/home/observer
[2023-06-13 13:39:17.906] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2023-06-13 13:39:17.906] [INFO] Trace ID: 86e80ca2-09ac-11ee-b959-fa163e4bac35
[2023-06-13 13:39:17.906] [INFO] If you want to view detailed obd logs, please run: obd display-trace 86e80ca2-09ac-11ee-b959-fa163e4bac35
[2023-06-13 13:39:17.906] [INFO]
看日志是ssh执行
cd ${source} && find -type f | xargs -i %(install_cmd)s ${source}/{} ${target}/{}” % {“install_cmd”: install_cmd}) 时超时。你手动ssh确认没问题的话 可以在配置里设置ssh timeout ，这里配置大一点 例如300 然后试一下
如果设置timeout之后不好使的话 可以手动执行一下 如下命令 看看卡在什么位置
export source=’/home/admin/.obd/repository/oceanbase-ce/4.1.0.0/21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe/admin’
export target=’/app/oceanbase/home/observer/admin’
cd ${source} && find -type f | xargs -i ln -fs ${source}/{} ${target}/{}
没有配置dns或者/etc/hosts没有配置域名对应地址，发不通
这是ob升级过程中内部报的错，我应该如何处理。
可以了，把3个节点的observer进程停掉，就可以正常更新了。