2025-08-05 08:22:41.399 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-08-05 08:22:41.412 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=9019461, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/redo/tst-ob1, primary_root_sys_password=xxx task_instance_id=9019387, task_operation=retry, ob_run_path=/opt/oceanbase/oceanbase, ob_install_path=/opt/oceanbase/oceanbase, ob_sql_port=2881, cluster_id=2000005, ocpagent_service_name=agent, ob_disk_path_style=DEFAULT, ob_svr_port=2882, operating_system=el8, sub_task_instance_name=Execute upgrade health checker script, watched_scenario=UPGRADE, sub_task_instance_id=9019461, primary_cluster_id=2000005, cluster_version=4.2.5.1, cluster_name=tst_ob1, mix_rpm_names={"zone02":"oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm","zone03":"oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm","zone01":"oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm"}, subtask_splitter=primary_first_cluster_ids, tst_ob1.1747712074.watched_parameters={"server_permanent_offline_time":"3600s","enable_rebalance":"{\"bill99\":\"True\",\"mtt\":\"True\",\"sys\":\"True\"}","enable_rereplication":"True"}, ob_cluster_id=1747712074, target_version=4.2.5.5-105000032025071717, dep_rpm_names={"zone02":["oceanbase-ce-libs-4.2.5.5-105000032025071717.el8.x86_64.rpm","oceanbase-ce-utils-4.2.5.5-105000032025071717.el8.x86_64.rpm"],"zone03":["oceanbase-ce-libs-4.2.5.5-105000032025071717.el8.x86_64.rpm","oceanbase-ce-utils-4.2.5.5-105000032025071717.el8.x86_64.rpm"],"zone01":["oceanbase-ce-libs-4.2.5.5-105000032025071717.el8.x86_64.rpm","oceanbase-ce-utils-4.2.5.5-105000032025071717.el8.x86_64.rpm"]}, target_cluster_status=RUNNING, ob_run_user=admin, latest_execution_start_time=2025-08-05T08:22:41.337+08:00, ob_data_disk_path=/data/tst-ob1}, listMap={standby_cluster_ids=[], 1747712074.tst_ob1.zone03.host_ids=[3000029], 1747712074.tst_ob1.zone_names=[zone01, zone02, zone03], 1747712074.tst_ob1.zone02.server_ids=[2000015], cluster_ids=[2000005], primary_first_cluster_ids =[2000005], 1747712074.tst_ob1.zone01.server_ids=[2000013], standby_first_cluster_ids=[2000005], 1747712074.tst_ob1.zone03.server_ids=[2000014], staged_versions=[4.2.5.5-105000032025071717], 1747712074.tst_ob1.zone01.host_ids=[3000027], direct_upgrade_versions=[], 1747712074.tst_ob1.zone02.host_ids=[3000028], zone_names=[zone01, zone02, zone03], server_ids=[2000013, 2000014, 2000015], host_ids=[3000027, 3000029, 3000028]}}, executor=10.10.75.155 2025-08-05 08:22:41.568 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=tst_ob1, tenantName=sys, dbUser=root 2025-08-05 08:22:41.578 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-08-05 08:22:41.587 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `svr_ip`, `svr_port`, `zone`, `role` FROM `CDB_OB_LS_LOCATIONS` WHERE `tenant_id` = 1 AND `ls_id` = 1 2025-08-05 08:22:41.688 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-08-05 08:22:41.698 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select version() 2025-08-05 08:22:41.874 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=10.10.75.33, port=2881, database=oceanbase) 2025-08-05 08:22:41.881 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=10.10.75.33, port=2881, obsAddrList=null, username=root, tenantName=sys, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-08-05 08:22:41.902 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-08-05 08:22:41.911 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-08-05 08:22:41.918 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-08-05 08:22:41.935 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.s.t.b.c.helper.UpgradeTaskHelper : execute script upgrade_health_checker.py on cluster, clusterId=2000005, obClusterId=1747712074, zone=null, clusterType=PRIMARY, rpmName=oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm 2025-08-05 08:22:41.961 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3000029 2025-08-05 08:22:41.976 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3000029 2025-08-05 08:22:42.096 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/file/exists, request body:GetFileExistsRequest(filePath=rpms/extract/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm/home/admin/oceanbase/etc/upgrade_health_checker.py), params:null 2025-08-05 08:22:42.152 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.h.s.HostOperationServiceImpl : [downloadRpm]: generate agent async task token:67b8930d-9a81-4882-a3bd-3f4631ca895c 2025-08-05 08:22:42.182 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3000029 2025-08-05 08:22:42.198 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3000029 2025-08-05 08:22:42.323 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/file/download, request body:DownloadFileRequest(source=RemoteFileAccessInfo(type=URL, url=http://10.10.75.150:3350/api/v2/object-storage/sys-package/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm, path=null), target=rpms/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm, validateChecksum=true, checksum=7c827b0fa70b06fcfcaaf0ebcacd4c793e9ea764), params:null 2025-08-05 08:22:42.343 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=67b8930d-9a81-4882-a3bd-3f4631ca895c), params:null 2025-08-05 08:22:42.354 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2025-08-05 08:22:42.362 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.common.lang.pattern.Retry : wait for 3 seconds 2025-08-05 08:22:45.371 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=67b8930d-9a81-4882-a3bd-3f4631ca895c), params:null 2025-08-05 08:22:45.391 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"path":"rpms/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm"},, 2025-08-05 08:22:45.413 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.h.s.HostOperationServiceImpl : [extractFileInRpm]: generate agent async task token:ffeacdc7-bfa9-4a74-9de4-b41e17ff59a9 2025-08-05 08:22:45.449 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3000029 2025-08-05 08:22:45.471 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3000029 2025-08-05 08:22:45.624 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/package/extract, request body:ExtractPackageRequest(packageFile=rpms/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm, targetPath=rpms/extract/, extractAll=true, fileInPackage=null), params:null 2025-08-05 08:22:45.657 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=ffeacdc7-bfa9-4a74-9de4-b41e17ff59a9), params:null 2025-08-05 08:22:45.672 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2025-08-05 08:22:45.679 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.common.lang.pattern.Retry : wait for 10 seconds 2025-08-05 08:22:55.686 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=ffeacdc7-bfa9-4a74-9de4-b41e17ff59a9), params:null 2025-08-05 08:22:55.706 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2025-08-05 08:22:55.716 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.common.lang.pattern.Retry : wait for 10 seconds 2025-08-05 08:23:05.724 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=ffeacdc7-bfa9-4a74-9de4-b41e17ff59a9), params:null 2025-08-05 08:23:05.747 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2025-08-05 08:23:05.757 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.common.lang.pattern.Retry : wait for 10 seconds 2025-08-05 08:23:15.764 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=ffeacdc7-bfa9-4a74-9de4-b41e17ff59a9), params:null 2025-08-05 08:23:15.776 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"basePath":"rpms/extract/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm","extractAll":true},, 2025-08-05 08:23:15.782 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.h.s.HostOperationServiceImpl : remove host files, pathList:[rpms/extract/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm/home/admin/oceanbase/admin, rpms/extract/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm/home/admin/oceanbase/bin, rpms/extract/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm/home/admin/oceanbase/etc/dict] 2025-08-05 08:23:15.807 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3000029 2025-08-05 08:23:15.819 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3000029 2025-08-05 08:23:15.922 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/file/remove, request body:RemoveFilesRequest(pathList=[rpms/extract/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm/home/admin/oceanbase/admin, rpms/extract/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm/home/admin/oceanbase/bin, rpms/extract/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm/home/admin/oceanbase/etc/dict]), params:null 2025-08-05 08:23:16.031 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.h.s.HostOperationServiceImpl : remove host files success, pathList:[rpms/extract/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm/home/admin/oceanbase/admin, rpms/extract/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm/home/admin/oceanbase/bin, rpms/extract/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm/home/admin/oceanbase/etc/dict] 2025-08-05 08:23:16.045 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.h.s.HostOperationServiceImpl : remove host files, pathList:[rpms/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm] 2025-08-05 08:23:16.077 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3000029 2025-08-05 08:23:16.096 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3000029 2025-08-05 08:23:16.269 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/file/remove, request body:RemoveFilesRequest(pathList=[rpms/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm]), params:null 2025-08-05 08:23:16.303 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.h.s.HostOperationServiceImpl : remove host files success, pathList:[rpms/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm] 2025-08-05 08:23:16.323 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.s.t.b.c.helper.UpgradeTaskHelper : [runScript]: generate agent async task token:b8d4f221-6241-439e-ba2b-1887d930e15e 2025-08-05 08:23:16.355 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3000029 2025-08-05 08:23:16.368 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3000029 2025-08-05 08:23:16.485 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/ob/upgrade/runScript, request body:RunUpgradeScriptRequest(scriptPath=rpms/extract/oceanbase-ce-4.2.5.5-105000032025071717.el8.x86_64.rpm/home/admin/oceanbase/etc/upgrade_health_checker.py, arguments=RunUpgradeScriptArgument(host=127.0.0.1, port=2881, logFile=4.2.5.5-105000032025071717_upgrade_health_checker_20250805082316.log, zone=null, cmdTimeout=0)), params:null 2025-08-05 08:23:16.510 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=b8d4f221-6241-439e-ba2b-1887d930e15e), params:null 2025-08-05 08:23:16.528 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2025-08-05 08:23:16.540 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.ocp.common.lang.pattern.Retry : wait for 15 seconds 2025-08-05 08:23:31.550 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.75.33:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=b8d4f221-6241-439e-ba2b-1887d930e15e), params:null 2025-08-05 08:23:31.562 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,null,, 2025-08-05 08:23:31.569 INFO 83331 --- [manual-subtask-executor10,64ecdf4f26ab0821,8436d7bd6ffdbddf] c.o.o.s.t.b.c.helper.UpgradeTaskHelper : execute script upgrade_health_checker.py finished Set state for subtask: 9019461, operation:RETRY, state: SUCCESSFUL Set state for subtask: 9019461, operation:RETRY, state: FAILED