2025-03-23 07:36:50.568 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-03-23 07:36:50.576 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=23000763, context=Context{parallelIdx=0, stringMap={cluster_name=HNYSOceanBase, target_server_status=RUNNING, subtask_splitter=1696919312.HNYSOceanBase.OceanBase01.server_ids, former_cluster_status=RUNNING, target_zone_status=RUNNING, primary_root_sys_password=xxx task_instance_id=23000704, task_operation=execute, zone_name=OceanBase01, ob_cluster_id=1696919312, cluster_id=2, ocpagent_service_name=agent, target_cluster_status=RUNNING, latest_execution_start_time=2025-03-23T07:36:50.546+08:00, sub_task_instance_name=Stop observer process, sub_task_instance_id=23000763}, listMap={1696919312.HNYSOceanBase.OceanBase01.server_ids=[3], 1696919312.HNYSOceanBase.OceanBase01.host_ids=[3], 1696919312.HNYSOceanBase.OceanBase02.server_ids=[4], 1696919312.HNYSOceanBase.OceanBase03.server_ids=[2], 1696919312.HNYSOceanBase.OceanBase02.host_ids=[4], 1696919312.HNYSOceanBase.OceanBase03.host_ids=[2], zone_names=[OceanBase01, OceanBase02, OceanBase03], host_ids=[3], server_ids=[3]}}, executor=10.32.103.51 2025-03-23 07:36:50.640 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.s.t.b.s.StopObServerProcessTask : begin to kill observer process, clusterId=2, obClusterId=1696919312, clusterType=PRIMARY, serverIp=10.32.103.100 2025-03-23 07:36:50.669 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-03-23 07:36:50.699 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-03-23 07:36:50.784 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.32.103.100:62888/api/v1/process/exists, request body:CheckProcessExistsRequest(name=observer), params:null 2025-03-23 07:36:50.810 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.s.t.b.server.ObServerTaskHandler : try to kill observer process,address is 10.32.103.100 2025-03-23 07:36:50.828 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-03-23 07:36:50.835 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-03-23 07:36:50.861 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.32.103.100:62888/api/v1/process/stop, request body:StopProcessRequest(process=FindProcessParam(findType=BY_NAME, name=observer, keyword=null, pid=null), force=true), params:null 2025-03-23 07:36:50.926 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.s.t.b.server.ObServerTaskHandler : wait for process observer to stop, waitTimes=1 2025-03-23 07:36:50.934 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] com.oceanbase.ocp.core.util.TaskUtils : [TaskUtils] wait 20 seconds 2025-03-23 07:37:10.957 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-03-23 07:37:10.972 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-03-23 07:37:11.001 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.32.103.100:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=observer, skipPorts=false), params:null 2025-03-23 07:37:11.017 WARN 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.c.h.s.HostOperationServiceImpl : Process name: observer is not found in host 3 2025-03-23 07:37:11.030 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.s.t.b.server.ObServerTaskHandler : process observer has been killed 2025-03-23 07:37:11.038 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] c.o.o.s.t.b.s.StopObServerProcessTask : kill observer process success 2025-03-23 07:37:11.057 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7114cbc8685d3ade] com.oceanbase.ocp.core.util.TaskUtils : [TaskUtils] wait 3 seconds Set state for subtask: 23000763, operation:EXECUTE, state: SUCCESSFUL 2025-03-23 10:14:16.417 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7da1f1674f0d7078] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-03-23 10:14:16.426 WARN 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7da1f1674f0d7078] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=23000763, context=Context{parallelIdx=0, stringMap={cluster_name=HNYSOceanBase, target_server_status=RUNNING, subtask_splitter=1696919312.HNYSOceanBase.OceanBase01.server_ids, former_cluster_status=RUNNING, target_zone_status=RUNNING, primary_root_sys_password=xxx task_instance_id=23000704, task_operation=rollback, zone_name=OceanBase01, ob_cluster_id=1696919312, cluster_id=2, ocpagent_service_name=agent, target_cluster_status=RUNNING, latest_execution_start_time=2025-03-23T10:14:16.380+08:00, sub_task_instance_name=Stop observer process, sub_task_instance_id=23000763}, listMap={1696919312.HNYSOceanBase.OceanBase01.server_ids=[3], 1696919312.HNYSOceanBase.OceanBase01.host_ids=[3], 1696919312.HNYSOceanBase.OceanBase02.server_ids=[4], 1696919312.HNYSOceanBase.OceanBase03.server_ids=[2], 1696919312.HNYSOceanBase.OceanBase02.host_ids=[4], 1696919312.HNYSOceanBase.OceanBase03.host_ids=[2], zone_names=[OceanBase01, OceanBase02, OceanBase03], host_ids=[3], server_ids=[3]}}, operation=ROLLBACK, executor=10.32.103.51 2025-03-23 10:14:16.488 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7da1f1674f0d7078] c.o.o.s.t.b.s.StopObServerProcessTask : begin to start observer process, serverId=3 2025-03-23 10:14:16.541 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7da1f1674f0d7078] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-03-23 10:14:16.555 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7da1f1674f0d7078] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-03-23 10:14:16.635 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7da1f1674f0d7078] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.32.103.100:62888/api/v1/process/exists, request body:CheckProcessExistsRequest(name=observer), params:null 2025-03-23 10:14:16.655 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7da1f1674f0d7078] c.o.o.s.t.b.server.ObServerTaskHandler : begin to start observer process, host ip is 10.32.103.100 2025-03-23 10:14:16.673 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7da1f1674f0d7078] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-03-23 10:14:16.685 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7da1f1674f0d7078] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-03-23 10:14:16.713 INFO 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7da1f1674f0d7078] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.32.103.100:62888/api/v1/ob/observer/start, request body:StartObServerProcessRequest(obPath=ObPath(installPath=/home/admin/oceanbase/HNYSOceanBase/oceanbase, dataPath=null, logPath=null, diskPathStyle=null, runPath=/home/admin/oceanbase/HNYSOceanBase/oceanbase), withArguments=false, arguments=null, runUser=admin, runGroup=null, readOnly=false), params:null 2025-03-23 10:14:16.923 ERROR 111533 --- [manual-subtask-executor14,2f06b1e93385a0d9,7da1f1674f0d7078] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [AgentClient]:http request is failed, response:Execute shell command failed: failed to execute command: Command{user=admin, program=sh, outputType=combined, cmd=cd /home/admin/oceanbase/HNYSOceanBase/oceanbase; ulimit -s 10240 -u 655350 -n 655350 -c unlimited; LD_LIBRARY_PATH=/home/admin/oceanbase/HNYSOceanBase/oceanbase/lib:$LD_LIBRARY_PATH LD_PRELOAD='' /home/admin/oceanbase/HNYSOceanBase/oceanbase/bin/observer, timeout=10s}, exitCode: 127, output: sh: line 0: cd: /home/admin/oceanbase/HNYSOceanBase/oceanbase: No such file or directory sh: /home/admin/oceanbase/HNYSOceanBase/oceanbase/bin/observer: No such file or directory , error: exit status 127 com.oceanbase.ocp.executor.exception.HttpRequestFailedException: [AgentClient]:http request is failed, response:Execute shell command failed: failed to execute command: Command{user=admin, program=sh, outputType=combined, cmd=cd /home/admin/oceanbase/HNYSOceanBase/oceanbase; ulimit -s 10240 -u 655350 -n 655350 -c unlimited; LD_LIBRARY_PATH=/home/admin/oceanbase/HNYSOceanBase/oceanbase/lib:$LD_LIBRARY_PATH LD_PRELOAD='' /home/admin/oceanbase/HNYSOceanBase/oceanbase/bin/observer, timeout=10s}, exitCode: 127, output: sh: line 0: cd: /home/admin/oceanbase/HNYSOceanBase/oceanbase: No such file or directory sh: /home/admin/oceanbase/HNYSOceanBase/oceanbase/bin/observer: No such file or directory , error: exit status 127 at com.oceanbase.ocp.executor.internal.template.HttpTemplate.checkSuccess(HttpTemplate.java:476) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doPost(HttpTemplate.java:286) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.postForNone(HttpTemplate.java:135) at com.oceanbase.ocp.executor.executor.AgentExecutor.startObServerProcess(AgentExecutor.java:351) at com.oceanbase.ocp.obops.internal.cluster.ClusterHostOperationServiceImpl.startObServerProcess(ClusterHostOperationServiceImpl.ja va:317) at com.oceanbase.ocp.service.task.business.server.ObServerTaskHandler.startObServerProcess(ObServerTaskHandler.java:74) at com.oceanbase.ocp.service.task.business.server.StopObServerProcessTask.rollback(StopObServerProcessTask.java:82) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.rollback(JavaSubtaskRunner.java:84) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:38) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Set state for subtask: 23000763, operation:ROLLBACK, state: FAILED