2026-08-18 10:43:09.250 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2026-08-18 10:43:09.256 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=2001223, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/data/log1, primary_root_sys_password=xxx task_instance_id=2000814, task_operation=execute, ob_run_path=/home/admin/oceanbase, cluster_type=PRIMARY, ob_install_path=/home/admin/oceanbase, obproxy_proxyro_username=proxyro, ob_sql_port=2881, cluster_id=1000004, operation_scope=Cluster, ob_svr_port=2882, sub_task_instance_name=Bootstrap ob, zone2.idc=daxigou, sub_task_instance_id=2001223, cluster_version=4.6.0.0, cluster_name=shequ46, target_server_status=RUNNING, mix_rpm_names={"zone3":"oceanbase-ce-4.6.0.0-100000162026071511.el7.x86_64.rpm","zone2":"oceanbase-ce-4.6.0.0-100000162026071511.el7.x86_64.rpm","zone1":"oceanbase-ce-4.6.0.0-100000162026071511.el7.x86_64.rpm"}, startup_option_string=large_query_threshold=6000000000s,net_thread_count=4,enable_syslog_recycle=TRUE,datafile_size=30G,cache_wash_threshold=2GB,cpu_count=4,max_syslog_file_count=3,log_disk_size=40G,__min_full_resource_pool_memory=1073741824,system_memory=3G,enable_cgroup=false,stack_size=512K,memory_limit=10G,enable_record_trace_log=TRUE, subtask_splitter=host_ids, ob_version=4.6.0.0-100000162026071511, service_name=shequ46:1785404810, target_zone_status=RUNNING, zone3.idc=dianxin, ob_cluster_id=1785404810, zone1.idc=langcao, service_type=OB_CLUSTER, obproxy_proxyro_password=xxx root_sys_password=xxx dep_rpm_names={"zone3":["oceanbase-ce-libs-4.6.0.0-100000162026071511.el7.x86_64.rpm","oceanbase-ce-utils-4.6.0.0-100000162026071511.el7.x86_64.rpm"],"zone2":["oceanbase-ce-libs-4.6.0.0-100000162026071511.el7.x86_64.rpm","oceanbase-ce-utils-4.6.0.0-100000162026071511.el7.x86_64.rpm"],"zone1":["oceanbase-ce-libs-4.6.0.0-100000162026071511.el7.x86_64.rpm","oceanbase-ce-utils-4.6.0.0-100000162026071511.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2026-08-18T10:43:09.232868343+ 08:00, ob_data_disk_path=/data/1, mix_obs_rpm={}}, listMap={zone3.host_ids=[1000003], root_servers=[1000001, 1000002, 1000003], server_ids=[1000010, 1000011, 1000012], zone2.host_ids=[1000002], host_ids=[1000001, 1000002, 1000003], zone1.host_ids=[1000001], zone_names=[zone1, zone2, zone3]}}, executor=10.10.1.70 2026-08-18 10:43:09.336 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2026-08-18 10:43:09.345 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2026-08-18 10:43:09.371 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.1.83:62888/api/v1/ob/observer/start, request body:StartObServerProcessRequest(obPath=ObPath(installPath=/home/admin/oceanbase, dataPath=/data/1, logPath=/data/log1, diskPathStyle=null, runPath=/home/admin/oceanbase), withArguments=true, arguments=StartObServerProcessArguments(serverIp=10.10.1.83, ipAddressType=LOCAL_IP, zoneName=zone3, configUrl=http://10.10.1.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=shequ46, rsList=10.10.1.81:2882:2881;10.10.1.82:2882:2881;10.10.1.83:2882:2881, clusterName=shequ46, obClusterId=1785404810, sqlPort=2881, svrPort=2882, optionString=large_query_threshold=6000000000s,net_thread_count=4,enable_syslog_recycle=TRUE,datafile_size=30G,cache_wash_threshold=2GB,cpu_count=4,max_syslog_file_count=3,log_disk_size=40G,__min_full_resource_pool_memory=1073741824,system_memory=3G,enable_cgroup=false,stack_size=512K,memory_limit=10G,enable_record_trace_log=TRUE, additionalLibPath=null), runUser=admin, runGroup=null, readOnly=false, additionalLibPath=null), params:null 2026-08-18 10:43:14.295 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2026-08-18 10:43:14.303 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000002 2026-08-18 10:43:14.329 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.1.82:62888/api/v1/ob/observer/start, request body:StartObServerProcessRequest(obPath=ObPath(installPath=/home/admin/oceanbase, dataPath=/data/1, logPath=/data/log1, diskPathStyle=null, runPath=/home/admin/oceanbase), withArguments=true, arguments=StartObServerProcessArguments(serverIp=10.10.1.82, ipAddressType=LOCAL_IP, zoneName=zone2, configUrl=http://10.10.1.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=shequ46, rsList=10.10.1.81:2882:2881;10.10.1.82:2882:2881;10.10.1.83:2882:2881, clusterName=shequ46, obClusterId=1785404810, sqlPort=2881, svrPort=2882, optionString=large_query_threshold=6000000000s,net_thread_count=4,enable_syslog_recycle=TRUE,datafile_size=30G,cache_wash_threshold=2GB,cpu_count=4,max_syslog_file_count=3,log_disk_size=40G,__min_full_resource_pool_memory=1073741824,system_memory=3G,enable_cgroup=false,stack_size=512K,memory_limit=10G,enable_record_trace_log=TRUE, additionalLibPath=null), runUser=admin, runGroup=null, readOnly=false, additionalLibPath=null), params:null 2026-08-18 10:43:20.223 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2026-08-18 10:43:20.232 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2026-08-18 10:43:20.257 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.1.81:62888/api/v1/ob/observer/start, request body:StartObServerProcessRequest(obPath=ObPath(installPath=/home/admin/oceanbase, dataPath=/data/1, logPath=/data/log1, diskPathStyle=null, runPath=/home/admin/oceanbase), withArguments=true, arguments=StartObServerProcessArguments(serverIp=10.10.1.81, ipAddressType=LOCAL_IP, zoneName=zone1, configUrl=http://10.10.1.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=shequ46, rsList=10.10.1.81:2882:2881;10.10.1.82:2882:2881;10.10.1.83:2882:2881, clusterName=shequ46, obClusterId=1785404810, sqlPort=2881, svrPort=2882, optionString=large_query_threshold=6000000000s,net_thread_count=4,enable_syslog_recycle=TRUE,datafile_size=30G,cache_wash_threshold=2GB,cpu_count=4,max_syslog_file_count=3,log_disk_size=40G,__min_full_resource_pool_memory=1073741824,system_memory=3G,enable_cgroup=false,stack_size=512K,memory_limit=10G,enable_record_trace_log=TRUE, additionalLibPath=null), runUser=admin, runGroup=null, readOnly=false, additionalLibPath=null), params:null 2026-08-18 10:43:20.969 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.s.t.b.c.helper.ObServerTaskHelper : begin to check server accessible, hostIds=[1000001, 1000002, 1000003] 2026-08-18 10:43:20.994 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2026-08-18 10:43:21.004 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2026-08-18 10:43:21.028 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.1.81:62888/api/v1/ob/observer/access, request body:AccessObServerProcessRequest(ip=10.10.1.81, port=2881, username=root), params:null 2026-08-18 10:43:21.035 WARN 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.s.t.b.c.helper.ObServerTaskHelper : Failed to check observer accessible, reason:[AgentClient]:http request is failed, response:Unexpected error: dial tcp 127.0.0.1:2881: connect: connection refused, cause:null 2026-08-18 10:43:21.039 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.ocp.common.lang.pattern.Retry : wait for 5 seconds 2026-08-18 10:43:26.074 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2026-08-18 10:43:26.105 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2026-08-18 10:43:26.181 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.1.81:62888/api/v1/ob/observer/access, request body:AccessObServerProcessRequest(ip=10.10.1.81, port=2881, username=root), params:null 2026-08-18 10:43:26.225 WARN 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.s.t.b.c.helper.ObServerTaskHelper : Failed to check observer accessible, reason:[AgentClient]:http request is failed, response:Unexpected error: Error 5150 (HY000): Tenant not in this server, cause:null 2026-08-18 10:43:26.235 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.ocp.common.lang.pattern.Retry : wait for 5 seconds 2026-08-18 10:43:31.259 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2026-08-18 10:43:31.269 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2026-08-18 10:43:31.298 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.1.81:62888/api/v1/ob/observer/access, request body:AccessObServerProcessRequest(ip=10.10.1.81, port=2881, username=root), params:null 2026-08-18 10:43:31.342 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2026-08-18 10:43:31.351 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000002 2026-08-18 10:43:31.377 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.1.82:62888/api/v1/ob/observer/access, request body:AccessObServerProcessRequest(ip=10.10.1.82, port=2881, username=root), params:null 2026-08-18 10:43:31.472 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2026-08-18 10:43:31.482 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2026-08-18 10:43:31.508 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.1.83:62888/api/v1/ob/observer/access, request body:AccessObServerProcessRequest(ip=10.10.1.83, port=2881, username=root), params:null 2026-08-18 10:43:34.137 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.s.t.b.c.helper.ObServerTaskHelper : all ob server accessible 2026-08-18 10:43:34.148 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.s.t.b.cluster.BootStrapObTask : bootstrap request: BootstrapRequest(clusterType=PRIMARY, rsInfoList=[RsInfo(region=chongqing, zone=zone1, server=ServerInfo(ip=10.10.1.81, port=2882)), RsInfo(region=chongqing, zone=zone2, server=ServerInfo(ip=10.10.1.82, port=2882)), RsInfo(region=chengdu, zone=zone3, server=ServerInfo(ip=10.10.1.83, port=2882))], primaryInfo=null, sqlPort=2881, timeoutSeconds=1200) 2026-08-18 10:43:34.164 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2026-08-18 10:43:34.174 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2026-08-18 10:43:34.200 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.1.81:62888/api/v1/ob/bootstrap, request body:BootstrapRequest(clusterType=PRIMARY, rsInfoList=[RsInfo(region=chongqing, zone=zone1, server=ServerInfo(ip=10.10.1.81, port=2882)), RsInfo(region=chongqing, zone=zone2, server=ServerInfo(ip=10.10.1.82, port=2882)), RsInfo(region=chengdu, zone=zone3, server=ServerInfo(ip=10.10.1.83, port=2882))], primaryInfo=null, sqlPort=2881, timeoutSeconds=1200), params:null 2026-08-18 10:43:34.224 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.1.81:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=3746db74-e3ee-487d-97fd-839dfc640a59), params:null 2026-08-18 10:43:34.230 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2026-08-18 10:43:34.235 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.ocp.common.lang.pattern.Retry : wait for 15 seconds 2026-08-18 10:43:49.240 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.1.81:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=3746db74-e3ee-487d-97fd-839dfc640a59), params:null 2026-08-18 10:43:49.248 INFO 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,OcpAgentError: code = 1002, message = Unexpected error: Error 4013 (HY001): No memory or reach tenant memory limit, 2026-08-18 10:43:49.254 ERROR 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.s.task.util.AgentAsyncTaskHelper : failed to execute async task, task token: 3746db74-e3ee-487d-97fd-839dfc640a59, err: OcpAgentError: code = 1002, message = Unexpected error: Error 4013 (HY001): No memory or reach tenant memory limit 2026-08-18 10:43:49.258 ERROR 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.agent.task.failed, and args [3746db74-e3ee-487d-97fd-839dfc640a59, OcpAgentError: code = 1002, message = Unexpected error: Error 4013 (HY001): No memory or reach tenant memory limit] 2026-08-18 10:43:49.263 ERROR 18070 --- [manual-subtask-executor-14,8938086b854b5b80,007264fc11acdb3c] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : OCP-Agent task 3746db74-e3ee-487d-97fd-839dfc640a59 failed, error message: OcpAgentError: code = 1002, message = Unexpected error: Error 4013 (HY001): No memory or reach tenant memory limit com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=AGENT_TASK_FAILED, args=3746db74-e3ee-487d-97fd-839dfc640a59,OcpAgentError: code = 1002, message = Unexpected error: Error 4013 (HY001): No memory or reach tenant memory limit at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$10.exception(ErrorCodes.java:2198) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:2053) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:169) at com.oceanbase.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:162) at com.oceanbase.ocp.service.task.util.AgentAsyncTaskHelper.checkSuccess(AgentAsyncTaskHelper.java:283) at com.oceanbase.ocp.service.task.util.AgentAsyncTaskHelper.waitForExecuteFinish(AgentAsyncTaskHelper.java:229) at com.oceanbase.ocp.service.task.util.AgentAsyncTaskHelper.bootstrapOb(AgentAsyncTaskHelper.java:133) at com.oceanbase.ocp.service.task.business.cluster.BootStrapObTask.run(BootStrapObTask.java:172) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:204) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:198) at com.oceanbase.ocp.core.task.engi ne.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$0(SubtaskExecutor.java:124) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:30) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Set state for subtask: 2001223, operation:EXECUTE, state: FAILED