2025-08-28 15:19:26.026 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-08-28 15:19:26.071 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1037787, context=Context{parallelIdx=0, stringMap={cluster_version=4.3.5.1, cluster_name=myobdemo, subtask_splitter=primary_first_cluster_ids, primary_root_sys_password=xxx task_instance_id=1037815, task_operation=execute, target_version=4.3.5.3-103000092025080818, cluster_id=2, ocpagent_service_name=agent, myobdemo.1744871768.watched_parameters={"server_permanent_offline_time":"3600s","enable_rebalance":"{\"bk_test\":\"True\",\"dsp_tt\":\"True\",\"oms_tt\":\"True\",\"sys\":\"True\"}","enable_rereplication":"True"}, operating_system=el7, target_cluster_status=RUNNING, latest_execution_start_time=2025-08-28T15:19:25.966+08:00, sub_task_instance_name=Execute upgrade checker script, watched_scenario=UPGRADE, sub_task_instance_id=1037787, primary_cluster_id=2}, listMap={standby_cluster_ids=[], 1744871768.myobdemo.zone1.server_ids=[2], 1744871768.myobdemo.zone2.host_ids=[3], 1744871768.myobdemo.zone5.host_ids=[7], cluster_ids=[2], 1744871768.myobdemo.zone4.server_ids=[1000001], primary_first_cluster_ids=[2], 1744871768.myobdemo.zone2.server_ids=[3], 1744871768.myobdemo.zone3.host_ids=[5], standby_first_cluster_ids=[2], 1744871768.myobdemo.zone5.server_ids=[1000005], 1744871768.myobdemo.zone1.host_ids=[2], 1744871768.myobdemo.zone3.server_ids=[1000007], staged_versions=[4.3.5.3-103000092025080818], 1744871768.myobdemo.zone_names=[zone1, zone2, zone3, zone4, zone5], 1744871768.myobdemo.zone4.host_ids=[6], direct_upgrade_versions=[]}}, executor=10.16.12.10 2025-08-28 15:19:26.136 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] invalidate ob connector, clusterName=myobdemo 2025-08-28 15:19:26.168 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] invalidate ob connector, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=dsp_tt, username=root, address=10.16.11.2, port=2881, database=oceanbase) 2025-08-28 15:19:26.174 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] ob connector is evicted from holder's cache. cause=EXPLICIT, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=dsp_tt, username=root, address=10.16.11.2, port=2881, database=oceanbase), size=9 2025-08-28 15:19:26.205 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] com.alibaba.druid.pool.DruidDataSource : {dataSource-8} closing ... 2025-08-28 15:19:26.239 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] com.alibaba.druid.pool.DruidDataSource : {dataSource-8} closed 2025-08-28 15:19:26.249 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] invalidate ob connector, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=oms_tt, username=root, address=10.16.11.2, port=2881, database=oceanbase) 2025-08-28 15:19:26.256 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] ob connector is evicted from holder's cache. cause=EXPLICIT, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=oms_tt, username=root, address=10.16.11.2, port=2881, database=oceanbase), size=8 2025-08-28 15:19:26.265 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] com.alibaba.druid.pool.DruidDataSource : {dataSource-9} closing ... 2025-08-28 15:19:26.280 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] com.alibaba.druid.pool.DruidDataSource : {dataSource-9} closed 2025-08-28 15:19:26.288 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] invalidate ob connector, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=bk_test, username=root, address=10.16.11.2, port=2881, database=oceanbase) 2025-08-28 15:19:26.309 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] ob connector is evicted from holder's cache. cause=EXPLICIT, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=bk_test, username=root, address=10.16.11.2, port=2881, database=oceanbase), size=7 2025-08-28 15:19:26.317 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] com.alibaba.druid.pool.DruidDataSource : {dataSource-7} closing ... 2025-08-28 15:19:26.341 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] com.alibaba.druid.pool.DruidDataSource : {dataSource-7} closed 2025-08-28 15:19:26.348 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] invalidate ob connector, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=sys, username=ocp_monitor, address=10.16.11.2, port=2881, database=oceanbase) 2025-08-28 15:19:26.355 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] ob connector is evicted from holder's cache. cause=EXPLICIT, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=sys, username=ocp_monitor, address=10.16.11.2, port=2881, database=oceanbase), size=6 2025-08-28 15:19:26.364 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] com.alibaba.druid.pool.DruidDataSource : {dataSource-25} closing ... 2025-08-28 15:19:26.389 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] com.alibaba.druid.pool.DruidDataSource : {dataSource-25} closed 2025-08-28 15:19:26.396 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] invalidate ob connector, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=sys, username=root, address=10.16.11.2, port=2881, database=oceanbase) 2025-08-28 15:19:26.403 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] ob connector is evicted from holder's cache. cause=EXPLICIT, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=sys, username=root, address=10.16.11.2, port=2881, database=oceanbase), size=5 2025-08-28 15:19:26.430 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] com.alibaba.druid.pool.DruidDataSource : {dataSource-3} closing ... 2025-08-28 15:19:26.461 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] com.alibaba.druid.pool.DruidDataSource : {dataSource-3} closed 2025-08-28 15:19:26.479 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] o.o.s.t.b.c.ExecUpgradeCheckerScriptTask : execute upgrade_checker.py, clusterId=2, version=4.3.5.3-103000092025080818 2025-08-28 15:19:26.590 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=sys, username=root, address=10.16.11.2, port=2881, database=oceanbase) 2025-08-28 15:19:26.599 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=sys, username=root, address=10.16.11.3, port=2881, database=oceanbase) 2025-08-28 15:19:26.605 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=sys, username=root, address=10.16.11.5, port=2881, database=oceanbase) 2025-08-28 15:19:26.611 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=sys, username=root, address=10.16.11.6, port=2881, database=oceanbase) 2025-08-28 15:19:26.620 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myobdemo, obClusterId=1744871768, tenantName=sys, username=root, address=10.16.11.4, port=2881, database=oceanbase) 2025-08-28 15:19:26.626 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=null, port=null, obsAddrList=[ObServerAddr(address=10.16.11.2, port=2881), ObServerAddr(address=10.16.11.3, port=2881), ObServerAddr(address=10.16.11.5, port=2881), ObServerAddr(address=10.16.11.6, port=2881), ObServerAddr(address=10.16.11.4, port=2881)], username=root, tenantName=sys, clusterName=myobdemo, obClusterId=1744871768, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-08-28 15:19:26.650 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-08-28 15:19:26.700 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myobdemo, tenantName=sys, dbUser=root 2025-08-28 15:19:26.717 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-08-28 15:19:26.725 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `svr_ip`, `svr_port`, `zone`, `role` FROM `CDB_OB_LS_LOCATIONS` WHERE `tenant_id` = 1 AND `ls_id` = 1 2025-08-28 15:19:26.809 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-08-28 15:19:26.819 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select version() 2025-08-28 15:19:26.870 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=10.16.11.2, port=2881, database=oceanbase) 2025-08-28 15:19:26.877 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=10.16.11.2, port=2881, obsAddrList=null, username=root, tenantName=sys, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-08-28 15:19:26.897 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-08-28 15:19:26.946 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-08-28 15:19:26.955 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-08-28 15:19:26.977 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.s.t.b.c.helper.UpgradeTaskHelper : execute script upgrade_checker.py on cluster, clusterId=2, obClusterId=1744871768, zone=null, clusterType=PRIMARY, rpmName=oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm 2025-08-28 15:19:27.014 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-08-28 15:19:27.030 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-08-28 15:19:27.283 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/file/exists, request body:GetFileExistsRequest(filePath=rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py), params:null 2025-08-28 15:19:27.347 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.h.s.HostOperationServiceImpl : [downloadRpm]: generate agent async task token:3e599a76-d17a-4767-8d1f-7925f4945212 2025-08-28 15:19:27.382 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-08-28 15:19:27.402 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-08-28 15:19:27.640 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/file/download, request body:DownloadFileRequest(source=RemoteFileAccessInfo(type=URL, url=http://10.16.12.10:8080/api/v2/object-storage/sys-package/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm, path=null), target=rpms/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm, validateChecksum=true, checksum=336e9664edf0eccf84ebe53a189e3445b60d7f7a), params:null 2025-08-28 15:19:27.666 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=3e599a76-d17a-4767-8d1f-7925f4945212), params:null 2025-08-28 15:19:27.675 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2025-08-28 15:19:27.688 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.common.lang.pattern.Retry : wait for 3 seconds 2025-08-28 15:19:30.696 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=3e599a76-d17a-4767-8d1f-7925f4945212), params:null 2025-08-28 15:19:30.723 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"path":"rpms/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm"},, 2025-08-28 15:19:30.732 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.h.s.HostOperationServiceImpl : [extractFileInRpm]: generate agent async task token:101090d3-3c4c-4bdd-8d7a-b76d9b878612 2025-08-28 15:19:30.773 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-08-28 15:19:30.801 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-08-28 15:19:30.997 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/package/extract, request body:ExtractPackageRequest(packageFile=rpms/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm, targetPath=rpms/extract/, extractAll=true, fileInPackage=null), params:null 2025-08-28 15:19:31.047 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=101090d3-3c4c-4bdd-8d7a-b76d9b878612), params:null 2025-08-28 15:19:31.120 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2025-08-28 15:19:31.128 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.common.lang.pattern.Retry : wait for 10 seconds 2025-08-28 15:19:41.138 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=101090d3-3c4c-4bdd-8d7a-b76d9b878612), params:null 2025-08-28 15:19:41.164 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2025-08-28 15:19:41.176 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.common.lang.pattern.Retry : wait for 10 seconds 2025-08-28 15:19:51.184 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=101090d3-3c4c-4bdd-8d7a-b76d9b878612), params:null 2025-08-28 15:19:51.206 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2025-08-28 15:19:51.214 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.common.lang.pattern.Retry : wait for 10 seconds 2025-08-28 15:20:01.223 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=101090d3-3c4c-4bdd-8d7a-b76d9b878612), params:null 2025-08-28 15:20:01.260 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"basePath":"rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm","extractAll":true},, 2025-08-28 15:20:01.338 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.h.s.HostOperationServiceImpl : remove host files, pathList:[rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/admin, rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/bin, rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/dict] 2025-08-28 15:20:01.379 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-08-28 15:20:01.424 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-08-28 15:20:01.690 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/file/remove, request body:RemoveFilesRequest(pathList=[rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/admin, rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/bin, rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/dict]), params:null 2025-08-28 15:20:01.937 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.h.s.HostOperationServiceImpl : remove host files success, pathList:[rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/admin, rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/bin, rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/dict] 2025-08-28 15:20:01.961 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.h.s.HostOperationServiceImpl : remove host files, pathList:[rpms/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm] 2025-08-28 15:20:02.118 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-08-28 15:20:02.250 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-08-28 15:20:02.494 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/file/remove, request body:RemoveFilesRequest(pathList=[rpms/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm]), params:null 2025-08-28 15:20:02.665 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.h.s.HostOperationServiceImpl : remove host files success, pathList:[rpms/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm] 2025-08-28 15:20:02.696 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.s.t.b.c.helper.UpgradeTaskHelper : [runScript]: generate agent async task token:8585143d-f9a5-4816-b9b8-c3d61df6cec7 2025-08-28 15:20:02.741 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-08-28 15:20:02.766 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-08-28 15:20:02.982 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/ob/upgrade/runScript, request body:RunUpgradeScriptRequest(scriptPath=rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py, arguments=RunUpgradeScriptArgument(host=127.0.0.1, port=2881, logFile=4.3.5.3-103000092025080818_upgrade_checker_20250828152002.log, zone=null, cmdTimeout=0)), params:null 2025-08-28 15:20:03.062 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=8585143d-f9a5-4816-b9b8-c3d61df6cec7), params:null 2025-08-28 15:20:03.214 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2025-08-28 15:20:03.346 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.common.lang.pattern.Retry : wait for 15 seconds 2025-08-28 15:20:18.367 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.16.11.2:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=8585143d-f9a5-4816-b9b8-c3d61df6cec7), params:null 2025-08-28 15:20:18.392 INFO 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,OcpAgentError: code = 1500, message = Execute shell command failed: failed to execute command: Command{user=root, program=sh, outputType=combined, cmd=export PYTHONPATH=/home/admin/ocp_agent/site-packages:$PYTHONPATH && python /tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py -h127.0.0.1 -P2881 -uroot@sys --password=xxx -l /tmp/4.3.5.3-103000092025080818_upgrade_checker_20250828152002.log, timeout=30m0s}, exitCode: 1, output: [2025-08-28 15:20:03] INFO upgrade_checker.py:1066 parameters from cmd: host="127.0.0.1", port=2881, user="root@sys", password=xxx timeout="0", log-file="/tmp/4.3.5.3-103000092025080818_upgrade_checker_20250828152002.log" [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:348 check observer version success, version = 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from oceanbase.__all_virtual_tenant_parameter_info where name='compatible', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.__all_tenant, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from __all_virtual_core_table where column_name in ('target_data_version', 'current_data_version') and column_value = 17180067073, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:399 check data version success, all tenant's compatible/target_data_version/current_data_vers ion is 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from __all_virtual_core_table where column_name in ('upgrade_begin_data_version') and column_value = 17180067073, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:410 check data version success, all tenant's upgrade_begin_data_version is 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as unsync_cnt from GV$OB_LOG_STAT where in_sync = 'NO', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:419 check paxos replica success [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from DBA_OB_TENANT_JOBS where job_status='INPROGRESS' and result_code is null, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as rebalance_task_cnt from CDB_OB_LS_REPLICA_TASKS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:431 check rebalance task success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_MAJOR_COMPACTION where (GLOBAL_BROADCAST_SCN > LAST_SCN or STATUS != 'IDLE'), rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select /*+ query_timeout(1000000000) */ count(1) from __all_virtual_tablet_compaction_info where max_received_scn > finished_scn and max_received_scn > 0, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:442 check cluster status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) as count from DBA_OB_TENANTS where status != 'NORMAL', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:454 check tenant status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) as count from oceanbase.__all_virtual_tenant_info where tenant_role != 'PRIMARY' and tenant_role != 'STANDBY', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:464 check tenant info success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from DBA_OB_TENANTS where LOCKED = 'YES', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:473 check tenant lock status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.gv$ob_units a, oceanbase.__all_tenant_history b where b.is_deleted = 1 and a.tenant_id = b.tenant_id, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:482 check deleted tenant unit gc success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_RESTORE_PROGRESS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:492 check restore job success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_ddl_task_status, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:538 check ddl task execut status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_BACKUP_JOBS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:549 check backup job success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status != "active"), row count = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:628 check observer status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select if (a.cnt = b.cnt, 1, 0) as passed from (select count(*) as cnt from oceanbase.__all_virtual_server_schema_info where refreshed_schema_version > 1 and refreshed_schema_version % 8 = 0) as a join (select count(*) as cnt from oceanbase.__all_server join oceanbase.__all_tenant) as b, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:635 check schema status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct(substring_index(build_version, '_', 1)) from __all_server, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:621 check server version success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select tenant_name from oceanbase.DBA_OB_TENANTS, rowcount = 7 [2025-08-28 15:20:05] INFO upgrade_checker.py:645 check special tenant name success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_clone_job, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:683 check tenant clone job success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_tenant_snapshot where status!='NORMAL', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:703 check tenant snapshot task success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from oceanbase.__all_virtual_tenant_parameter_info where (name like "log_transport_compress_func" and value like "zlib_1.0"), rowcount = 1 [20 25-08-28 15:20:05] INFO upgrade_checker.py:651 check log_transport_compress_func success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select /*+ query_timeout(1000000000) */ count(1) from __all_virtual_table where (compress_func_name like '%zlib%'), rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:657 check table compression method success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from GV$OB_PARAMETERS where (name like "tableapi_transport_compress_func" and value like "zlib%");, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:663 check table_api_transport_compress_func success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:723 check variable binlog_row_image success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select tenant_id from oceanbase.__all_tenant where compatibility_mode = 1, rowcount = 0 [2025-08-28 15:20:05] INFO upgrade_checker.py:757 check oracle standby_replication privs success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:779 no need to check disk space, min observer version: 4.3.5.1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:960 no need to check cs encoding arch compatibility for cluster version after version 4.3.3 [2025-08-28 15:20:05] INFO upgrade_checker.py:985 check upgrade for arch-dependant cs_enco ding format success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_load_data_stat, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:917 check direct load task execut status success [2025-08-28 15:20:05] ERROR upgrade_checker.py:1034 run error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' [2025-08-28 15:20:05] ERROR upgrade_checker.py:1043 normal error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' [2025-08-28 15:20:05] ERROR upgrade_checker.py:1072 normal error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1067, in do_check(host, port, user, password, timeout, upgrade_params, cpu_arch) File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) MyError: 'upgr ade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1067, in do_check(host, port, user, password, timeout, upgrade_params, cpu_arch) File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) __main__.MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' , error: exit status 1, 2025-08-28 15:20:18.606 ERROR 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.s.task.util.AgentAsyncTaskHelper : failed to execute async task, task token: 8585143d-f9a5-4816-b9b8-c3d61df6cec7, err: OcpAgentError: code = 1500, message = Execute shell command failed: failed to execute command: Command{user=root, program=sh, outputType=combined, cmd=export PYTHONPATH=/home/admin/ocp_agent/site-packages:$PYTHONPATH && python /tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py -h127.0.0.1 -P2881 -uroot@sys --password=xxx -l /tmp/4.3.5.3-103000092025080818_upgrade_checker_20250828152002.log, timeout=30m0s}, exitCode: 1, output: [2025-08-28 15:20:03] INFO upgrade_checker.py:1066 parameters from cmd: host="127.0.0.1", port=2881, user="root@sys", password=xxx timeout="0", log-file="/tmp/4.3.5.3-103000092025080818_upgrade_checker_20250828152002.log" [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:348 check observer version success, version = 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from oceanbase.__all_virtual_tenant_parameter_info where name='compatible', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.__all_tenant, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from __all_virtual_core_table where column_name in ('target_data_version', 'current_data_version') and column_value = 17180067073, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:399 check data version success, all tenant's compatible/target_ data_version/current_data_version is 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from __all_virtual_core_table where column_name in ('upgrade_begin_data_version') and column_value = 17180067073, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:410 check data version success, all tenant's upgrade_begin_data_version is 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as unsync_cnt from GV$OB_LOG_STAT where in_sync = 'NO', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:419 check paxos replica success [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from DBA_OB_TENANT_JOBS where job_status='INPROGRESS' and result_code is null, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as rebalance_task_cnt from CDB_OB_LS_REPLICA_TASKS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:431 check rebalance task success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_MAJOR_COMPACTION where (GLOBAL_BROADCAST_SCN > LAST_SCN or STATUS != 'IDLE'), rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select /*+ query_timeout(1000000000) */ count(1) from __all_virtual_tablet_compaction_info where max_received_scn > finished_scn and max_received_scn > 0, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:442 check cluster status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) as count from DBA_OB_TENANTS where status != 'NORMAL', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:454 check tenant status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) as count from oceanbase.__all_virtual_tenant_info where tenant_role != 'PRIMARY' and tenant_role != 'STANDBY', rowcount = 1 [2025-08-28 15:20:04] INF O upgrade_checker.py:464 check tenant info success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from DBA_OB_TENANTS where LOCKED = 'YES', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:473 check tenant lock status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.gv$ob_units a, oceanbase.__all_tenant_history b where b.is_deleted = 1 and a.tenant_id = b.tenant_id, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:482 check deleted tenant unit gc success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_RESTORE_PROGRESS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:492 check restore job success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_ddl_task_status, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:538 check ddl task execut status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_BACKUP_JOBS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:549 check backup job success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status != "active"), rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:628 check observer status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select if (a.cnt = b.cnt, 1, 0) as passed from (select count(*) as cnt from oceanbase.__all_virtual_server_schema_info where refreshed_schema_version > 1 and refreshed_schema_version % 8 = 0) as a join (select count(*) as cnt from oceanbase.__all_server join oceanbase.__all_tenant) as b, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:635 check schema status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct(substring_index(build_version, '_', 1)) from __all_server, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:621 check server version success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select tenant_name from oceanbase.DBA_OB_TENANTS, rowcount = 7 [2025-08-28 15:20:05] INFO upgrade_checker.py:645 check special tenant name success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_clone_job, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:683 check tenant clone job success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_tenant_snapshot where status!='NORMAL', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:703 check tenant snapshot task success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from oceanbase.__all_virtual_tenant_parameter_info where (name like "log_transport_compress_func" and value like "zlib_1.0"), rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:651 check log_transport_compress_func success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select /*+ query_timeout(1000000000) */ count(1) from __all_virtual_table where (compress_func_name like '%zlib%'), rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:657 check table compression method success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from GV$OB_PARAMETERS where (name like "tableapi_transport_compress_func" and value like "zlib%");, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:663 check table_api_transport_compress_func success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:723 check variable binlog_row_image success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select tenant_id from oceanbase.__all_tenant where compatibility_mode = 1, rowcount = 0 [2025-08-28 15:20:05] INFO upgrade_checker.py:757 check oracle standby_replication privs success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:779 no need to check disk space, min observer version: 4.3.5.1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:960 no need to check cs encoding arch compatibility for cluster version after version 4.3.3 [2025-08-28 15:20:05] INFO upgrade_checker.py:985 check upgr ade for arch-dependant cs_encoding format success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_load_data_stat, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:917 check direct load task execut status success [2025-08-28 15:20:05] ERROR upgrade_checker.py:1034 run error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' [2025-08-28 15:20:05] ERROR upgrade_checker.py:1043 normal error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' [2025-08-28 15:20:05] ERROR upgrade_checker.py:1072 normal error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1067, in do_check(host, port, user, password, timeout, upgrade_params, cpu_arch) File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyE rror(error_msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1067, in do_check(host, port, user, password, timeout, upgrade_params, cpu_arch) File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) __main__.MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' , error: exit status 1 2025-08-28 15:20:18.760 ERROR 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.agent.task.failed, and args [8585143d-f9a5-4816-b9b8-c3d61df6cec7, OcpAgentError: code = 1500, message = Execute shell command failed: failed to execute command: Command{user=root, program=sh, outputType=combined, cmd=export PYTHONPATH=/home/admin/ocp_agent/site-packages:$PYTHONPATH && python /tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py -h127.0.0.1 -P2881 -uroot@sys --password=xxx -l /tmp/4.3.5.3-103000092025080818_upgrade_checker_20250828152002.log, timeout=30m0s}, exitCode: 1, output: [2025-08-28 15:20:03] INFO upgrade_checker.py:1066 parameters from cmd: host="127.0.0.1", port=2881, user="root@sys", password=xxx timeout="0", log-file="/tmp/4.3.5.3-103000092025080818_upgrade_checker_20250828152002.log" [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:348 check observer version success, version = 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from oceanbase.__all_virtual_tenant_parameter_info where name='compatible', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.__all_tenant, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from __all_virtual_core_table where column_name in ('target_data_version', 'current_data_version') and column_value = 17180067073, rowcount = 1 [2025-08-28 15:20:03] INFO upg rade_checker.py:399 check data version success, all tenant's compatible/target_data_version/current_data_version is 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from __all_virtual_core_table where column_name in ('upgrade_begin_data_version') and column_value = 17180067073, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:410 check data version success, all tenant's upgrade_begin_data_version is 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as unsync_cnt from GV$OB_LOG_STAT where in_sync = 'NO', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:419 check paxos replica success [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from DBA_OB_TENANT_JOBS where job_status='INPROGRESS' and result_code is null, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as rebalance_task_cnt from CDB_OB_LS_REPLICA_TASKS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:431 check rebalance task success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_MAJOR_COMPACTION where (GLOBAL_BROADCAST_SCN > LAST_SCN or STATUS != 'IDLE'), rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select /*+ query_timeout(1000000000) */ count(1) from __all_virtual_tablet_compaction_info where max_received_scn > finished_scn and max_received_scn > 0, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:442 check cluster status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) as count from DBA_OB_TENANTS where status != 'NORMAL', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:454 check tenant status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) as count from oceanbase.__all_virtual_tenant_info where tenant_role != 'PRIMARY' and tenant_role != 'STANDBY', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:464 check tenant info success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from DBA_OB_TENANTS where LOCKED = 'YES', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:473 check tenant lock status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.gv$ob_units a, oceanbase.__all_tenant_history b where b.is_deleted = 1 and a.tenant_id = b.tenant_id, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:482 check deleted tenant unit gc success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_RESTORE_PROGRESS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:492 check restore job success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_ddl_task_status, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:538 check ddl task execut status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_BACKUP_JOBS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:549 check backup job success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status != "active"), rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:628 check observer status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select if (a.cnt = b.cnt, 1, 0) as passed from (select count(*) as cnt from oceanbase.__all_virtual_server_schema_info where refreshed_schema_version > 1 and refreshed_schema_version % 8 = 0) as a join (select count(*) as cnt from oceanbase.__all_server join oceanbase.__all_tenant) as b, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:635 check schema status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct(substring_index(build_version, '_', 1)) from __all_server, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:621 check server version success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select tenant_name from oceanbase.DBA_OB_TENANTS, rowcount = 7 [2025-08-28 15:20:05] INFO upgrade_checker.py:645 check special tenant name success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_clone_job, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:683 check tenant clone job success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_tenant_snapshot where status!='NORMAL', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:703 check tenant snapshot task success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from oceanbase.__all_virtual_tena nt_parameter_info where (name like "log_transport_compress_func" and value like "zlib_1.0"), rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:651 check log_transport_compress_func success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select /*+ query_timeout(1000000000) */ count(1) from __all_virtual_table where (compress_func_name like '%zlib%'), rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:657 check table compression method success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from GV$OB_PARAMETERS where (name like "tableapi_transport_compress_func" and value like "zlib%");, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:663 check table_api_transport_compress_func success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:723 check variable binlog_row_image success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select tenant_id from oceanbase.__all_tenant where compatibility_mode = 1, rowcount = 0 [2025-08-28 15:20:05] INFO upgrade_checker.py:757 check oracle standby_replication privs success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:779 no need to check disk space, min observer version: 4.3.5.1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:960 no need to check cs encoding arch compatibility for cluster version a fter version 4.3.3 [2025-08-28 15:20:05] INFO upgrade_checker.py:985 check upgrade for arch-dependant cs_encoding format success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_load_data_stat, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:917 check direct load task execut status success [2025-08-28 15:20:05] ERROR upgrade_checker.py:1034 run error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' [2025-08-28 15:20:05] ERROR upgrade_checker.py:1043 normal error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' [2025-08-28 15:20:05] ERROR upgrade_checker.py:1072 normal error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1067, in do_check(host, port, user, password, timeout, upgrade_params, cpu_arch) File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admi n/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1067, in do_check(host, port, user, password, timeout, upgrade_params, cpu_arch) File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) __main__.MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' , error: exit status 1] 2025-08-28 15:20:18.917 ERROR 13042 --- [manual-subtask-executor16,5695bdf60600eedb,1471e9949ef1dc75] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : OCP-Agent task 8585143d-f9a5-4816-b9b8-c3d61df6cec7 failed, error message: OcpAgentError: code = 1500, message = Execute shell command failed: failed to execute command: Command{user=root, program=sh, outputType=combined, cmd=export PYTHONPATH=/home/admin/ocp_agent/site-packages:$PYTHONPATH && python /tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py -h127.0.0.1 -P2881 -uroot@sys --password=xxx -l /tmp/4.3.5.3-103000092025080818_upgrade_checker_20250828152002.log, timeout=30m0s}, exitCode: 1, output: [2025-08-28 15:20:03] INFO upgrade_checker.py:1066 parameters from cmd: host="127.0.0.1", port=2881, user="root@sys", password=xxx timeout="0", log-file="/tmp/4.3.5.3-103000092025080818_upgrade_checker_20250828152002.log" [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:348 check observer version success, version = 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from oceanbase.__all_virtual_tenant_parameter_info where name='compatible', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.__all_tenant, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from __all_virtual_core_table where column_name in ('target_data_version', 'current_data_version') and column_value = 17180067073, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:399 check data version success, all tenant's compatible/target_data_versi on/current_data_version is 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from __all_virtual_core_table where column_name in ('upgrade_begin_data_version') and column_value = 17180067073, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:410 check data version success, all tenant's upgrade_begin_data_version is 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as unsync_cnt from GV$OB_LOG_STAT where in_sync = 'NO', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:419 check paxos replica success [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from DBA_OB_TENANT_JOBS where job_status='INPROGRESS' and result_code is null, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as rebalance_task_cnt from CDB_OB_LS_REPLICA_TASKS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:431 check rebalance task success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_MAJOR_COMPACTION where (GLOBAL_BROADCAST_SCN > LAST_SCN or STATUS != 'IDLE'), rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select /*+ query_timeout(1000000000) */ count(1) from __all_virtual_tablet_compaction_info where max_received_scn > finished_scn and max_received_scn > 0, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:442 check cluster status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) as count from DBA_OB_TENANTS where status != 'NORMAL', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:454 check tenant status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) as count from oceanbase.__all_virtual_tenant_info where tenant_role != 'PRIMARY' and tenant_role != 'STANDBY', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_ checker.py:464 check tenant info success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from DBA_OB_TENANTS where LOCKED = 'YES', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:473 check tenant lock status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.gv$ob_units a, oceanbase.__all_tenant_history b where b.is_deleted = 1 and a.tenant_id = b.tenant_id, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:482 check deleted tenant unit gc success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_RESTORE_PROGRESS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:492 check restore job success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_ddl_task_status, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:538 check ddl task execut status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_BACKUP_JOBS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:549 check backup job success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or stat us != "active"), rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:628 check observer status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select if (a.cnt = b.cnt, 1, 0) as passed from (select count(*) as cnt from oceanbase.__all_virtual_server_schema_info where refreshed_schema_version > 1 and refreshed_schema_version % 8 = 0) as a join (select count(*) as cnt from oceanbase.__all_server join oceanbase.__all_tenant) as b, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:635 check schema status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct(substring_index(build_version, '_', 1)) from __all_server, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:621 check server version success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select tenant_name from oceanbase.DBA_OB_TENANTS, rowcount = 7 [2025-08-28 15:20:05] INFO upgrade_checker.py:645 check special tenant name success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_clone_job, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:683 check tenant clone job success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_tenant_snapshot where status!='NORMAL', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:703 check tenant snapshot task success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from oceanbase.__all_virtual_tenant_parameter_info where (name like "log_transport_compress_func" and value like "zlib_1.0 "), rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:651 check log_transport_compress_func success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select /*+ query_timeout(1000000000) */ count(1) from __all_virtual_table where (compress_func_name like '%zlib%'), rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:657 check table compression method success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from GV$OB_PARAMETERS where (name like "tableapi_transport_compress_func" and value like "zlib%");, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:663 check table_api_transport_compress_func success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:723 check variable binlog_row_image success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select tenant_id from oceanbase.__all_tenant where compatibility_mode = 1, rowcount = 0 [2025-08-28 15:20:05] INFO upgrade_checker.py:757 check oracle standby_replication privs success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:779 no need to check disk space, min observer version: 4.3.5.1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:960 no need to check cs encoding arch compatibility for cluster version after version 4.3.3 [2025-08-28 15:20:05] INFO upgrade_checker.py:985 check upgrade for ar ch-dependant cs_encoding format success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_load_data_stat, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:917 check direct load task execut status success [2025-08-28 15:20:05] ERROR upgrade_checker.py:1034 run error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' [2025-08-28 15:20:05] ERROR upgrade_checker.py:1043 normal error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' [2025-08-28 15:20:05] ERROR upgrade_checker.py:1072 normal error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1067, in do_check(host, port, user, password, timeout, upgrade_params, cpu_arch) File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error _msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1067, in do_check(host, port, user, password, timeout, upgrade_params, cpu_arch) File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) __main__.MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' , error: exit status 1 com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=AGENT_TASK_FAILED, args=8585143d-f9a5-4816-b9b8-c3d61df6cec7,OcpAgentError: code = 1500, message = Execute shell command failed: failed to execute command: Command{user=root, program=sh, outputType=combined, cmd=export PYTHONPATH=/home/admin/ocp_agent/site-packages:$PYTHONPATH && python /tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py -h127.0.0.1 -P2881 -uroot@sys --password=xxx -l /tmp/4.3.5.3-103000092025080818_upgrade_checker_20250828152002.log, timeout=30m0s}, exitCode: 1, output: [2025-08-28 15:20:03] INFO upgrade_checker.py:1066 parameters from cmd: host="127.0.0.1", port=2881, user="root@sys", password=xxx timeout="0", log-file="/tmp/4.3.5.3-103000092025080818_upgrade_checker_20250828152002.log" [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:348 check observer version success, version = 4.3.5.1 [2025-08-28 15:20:03] INFO upgra de_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from oceanbase.__all_virtual_tenant_parameter_info where name='compatible', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.__all_tenant, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from __all_virtual_core_table where column_name in ('target_data_version', 'current_data_version') and column_value = 17180067073, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:399 check data version success, all tenant's compatible/target_data_version/current_data_version is 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from __all_virtual_core_table where column_name in ('upgrade_begin_data_version') and column_value = 17180067073, rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:410 check data version success, all tenant's upgrade_begin_data_version is 4.3.5.1 [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as unsync_cnt from GV$OB_LOG_STAT where in_sync = 'NO', rowcount = 1 [2025-08-28 15:20:03] INFO upgrade_checker.py:419 check paxos replica success [2025-08-28 15:20:03] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from DBA_OB_TENANT_JOBS where job_status='INPROGRESS' and result_code is null, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as rebalance_task_cnt from CDB_OB_LS_REPLICA_TASKS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:431 check rebalance task success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_MAJOR_COMPACTION where (GLOBAL_BROADCAST_SCN > LAST_SCN or STATUS != 'IDLE'), rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select /*+ query_timeout(1000000000) */ count(1) from __all_virtual_tablet_compaction_info where max_received_scn > finished_scn and max_received_scn > 0, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:442 check cluster status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) as count from DBA_OB_TENANTS where status != 'NORMAL', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:454 check tenant status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) as count from oceanbase.__all_virtual_tenant_info where tenant_role != 'PRIMARY' and tenant_role != 'STANDBY', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:464 check tenant info success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from DBA_OB_TENANTS where LOCKED = 'YES', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:473 check tenant lock status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.gv$ob_units a, oceanbase.__all_tenant_history b where b.is_deleted = 1 and a.tenant_id = b.tenant_id, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:482 check deleted tenant unit gc success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_RESTORE_PROGRESS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:492 check restore job success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_ddl_task_status, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:538 check ddl task execut status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from CDB_OB_BACKUP_JOBS, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:549 check backup job success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status != "active"), rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:628 check observer status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select if (a.cnt = b.cnt, 1, 0) as passed from (select count(*) as cnt from oceanbase.__all_virtual_server_schema_info where refreshed_schema_version > 1 and refreshed_schema_version % 8 = 0) as a join (select count(*) as cnt from oceanbase.__all_server join oceanbase.__all_tenant) as b, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:635 check schema status success [2025-08-28 15:20:04] INFO upgrade_checker.py:58 succeed to execute query: select distinct(substring_index(build_version, '_', 1)) from __all_server, rowcount = 1 [2025-08-28 15:20:04] INFO upgrade_checker.py:621 check server version success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select tenant_name from oceanbase.DBA_OB_TENANTS, rowcount = 7 [2025-08-28 15:20:05] INFO upgrade_checker.py:645 check special tenant name success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker .py:58 succeed to execute query: select count(1) from __all_virtual_clone_job, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:683 check tenant clone job success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_tenant_snapshot where status!='NORMAL', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:703 check tenant snapshot task success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from oceanbase.__all_virtual_tenant_parameter_info where (name like "log_transport_compress_func" and value like "zlib_1.0"), rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:651 check log_transport_compress_func success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select /*+ query_timeout(1000000000) */ count(1) from __all_virtual_table where (compress_func_name like '%zlib%'), rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:657 check table compression method success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) as cnt from GV$OB_PARAMETERS where (name like "tableapi_transport_compress_func" and value like "zlib%");, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:663 check table_api_transport_compress_func success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:723 check variable binlog_row_image success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select tenant_id from ocean base.__all_tenant where compatibility_mode = 1, rowcount = 0 [2025-08-28 15:20:05] INFO upgrade_checker.py:757 check oracle standby_replication privs success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:779 no need to check disk space, min observer version: 4.3.5.1 [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select distinct value from GV$OB_PARAMETERS where name='min_observer_version', rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:960 no need to check cs encoding arch compatibility for cluster version after version 4.3.3 [2025-08-28 15:20:05] INFO upgrade_checker.py:985 check upgrade for arch-dependant cs_encoding format success [2025-08-28 15:20:05] INFO upgrade_checker.py:58 succeed to execute query: select count(1) from __all_virtual_load_data_stat, rowcount = 1 [2025-08-28 15:20:05] INFO upgrade_checker.py:917 check direct load task execut status success [2025-08-28 15:20:05] ERROR upgrade_checker.py:1034 run error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' [2025-08-28 15:20:05] ERROR upgrade_checker.py:1043 normal error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_ fail_list raise MyError(error_msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' [2025-08-28 15:20:05] ERROR upgrade_checker.py:1072 normal error Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1067, in do_check(host, port, user, password, timeout, upgrade_params, cpu_arch) File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' Traceback (most recent call last): File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1067, in do_check(host, port, user, password, timeout, upgrade_params, cpu_arch) File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 1031, in do_check check_fail_list() File "/tmp/rpms/extract/oceanbase-ce-4.3.5.3-103000092025080818.el7.x86_64.rpm/home/admin/oceanbase/etc/upgrade_checker.py", line 762, in check_fail_list raise MyError(error_msg) __main__.MyError: 'upgrade checker failed with 1 reasons: [1 locality tasks is doing, please check] ' , error: exit status 1 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$10.exception(ErrorCodes.java:1814) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1669) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:169) at com.oceanbase.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:162) at com.oceanbase.ocp.service.task.util.AgentAsyncTaskHelper.checkS uccess(AgentAsyncTaskHelper.java:279) at com.oceanbase.ocp.service.task.util.AgentAsyncTaskHelper.waitForExecuteFinish(AgentAsyncTaskHelper.java:225) at com.oceanbase.ocp.service.task.util.AgentAsyncTaskHelper.runUpgradeScript(AgentAsyncTaskHelper.java:140) at com.oceanbase.ocp.service.task.business.cluster.helper.UpgradeTaskHelper.runScript(UpgradeTaskHelper.java:161) at com.oceanbase.ocp.service.task.business.cluster.helper.UpgradeTaskHelper.runScript(UpgradeTaskHelper.java:106) at com.oceanbase.ocp.service.task.business.cluster.helper.UpgradeTaskHelper.runScript(UpgradeTaskHelper.java:81) at com.oceanbase.ocp.service.task.business.cluster.ExecUpgradeCheckerScriptTask.run(ExecUpgradeCheckerScriptTask.java:60) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Set state for subtask: 1037787, operation:EXECUTE, state: FAILED Set state for subtask: 1037787, operation:EXECUTE, state: FAILED 2025-08-28 15:49:37.470 WARN 13042 --- [subtask-executor28,89712dfa8ae7971e,45632b64c43c1f0e] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask timeout, name=Execute upgrade checker script, executedSeconds=1811, timeout=1800