2024-04-18 09:52:50.083 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=402, context=Context{parallelIdx=7, stringMap={cluster_version=4.2.2.0, cluster_name=core_cluster, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=45685, service_name=core_cluster:1713323324, target_zone_status=RUNNING, task_instance_id=348, ob_connect_address=10.191.69.51:2883, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.2.0, host_credential_id=4, cluster_id=2, root_sys_password=******, service_type=OB_CLUSTER, obproxy_proxyro_password=******, ob_data_dir=/data, connection_mode=proxy, target_cluster_status=RUNNING, latest_execution_start_time=2024-04-18T09:52:49.269+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=402}, listMap={add_region_ids=[], server_ids=[4, 5, 6, 7, 8, 9, 10, 11, 12, 13], add_idc_ids=[2, 3, 4], all_host_ids=[4, 5, 6, 7, 8, 9, 10, 11, 12, 13], add_host_ids=[4, 5, 6, 7, 8, 9, 10, 11, 12, 13], host_ids=[4, 5, 6, 7, 8, 9, 10, 11, 12, 13], zone_names=[zone4, zone1, zone3, zone2, zone5]}}, executor=10.191.69.58 2024-04-18 09:52:50.084 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-04-18 09:52:50.085 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=11 2024-04-18 09:52:50.088 WARN 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=11 2024-04-18 09:52:50.091 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=11 2024-04-18 09:52:50.095 WARN 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=11 2024-04-18 09:52:50.096 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-04-18 09:52:50.097 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-04-18 09:52:50.109 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:52:50.123 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:52:50.163 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:52:50.176 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:52:50.216 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'netstat -ant| awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"'' on 10.191.69.57 2024-04-18 09:52:50.241 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'netstat -ant| awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"'' on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=sudo bash -c 'netstat -ant| awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-04-18 09:52:50.242 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'netstat -ant| awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"'' on 10.191.69.57,result:SshResult(host=10.191.69.57, username=admin, command=sudo bash -c 'netstat -ant| awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-04-18 09:52:50.281 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:52:50.293 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:52:50.333 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'netstat -ant| awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"'' on 10.191.69.57 2024-04-18 09:52:50.357 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'netstat -ant| awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"'' on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=sudo bash -c 'netstat -ant| awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-04-18 09:52:50.358 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'netstat -ant| awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"'' on 10.191.69.57,result:SshResult(host=10.191.69.57, username=admin, command=sudo bash -c 'netstat -ant| awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-04-18 09:52:50.371 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-04-18 09:52:50.397 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:52:50.410 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:52:50.450 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' on 10.191.69.57 2024-04-18 09:52:50.476 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm', out=curl: (7) Failed connect to localhost:8080; Connection refused, err=, extOut=null, exitStatus=7) 2024-04-18 09:52:50.477 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' on 10.191.69.57,result:SshResult(host=10.191.69.57, username=admin, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm', out=curl: (7) Failed connect to localhost:8080; Connection refused, err=, extOut=null, exitStatus=7) 2024-04-18 09:52:50.478 INFO 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm', return code:7, output:curl: (7) Failed connect to localhost:8080; Connection refused, extOut:null 2024-04-18 09:52:50.480 ERROR 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm', exit code: 7, cause:null 2024-04-18 09:52:50.481 ERROR 22530 --- [pool-manual-subtask-executor16,0a1cccbb0e154590,520c161b8eef] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:399) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:383) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:74) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:55) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:382) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:228) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:219) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:704) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$6315daad.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallO cpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:203) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:197) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:134) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Set state for subtask: 402, operation:EXECUTE, state: FAILED 2024-04-18 09:54:33.057 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=402, context=Context{parallelIdx=7, stringMap={cluster_version=4.2.2.0, cluster_name=core_cluster, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=45685, service_name=core_cluster:1713323324, target_zone_status=RUNNING, task_instance_id=348, ob_connect_address=10.191.69.51:2883, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.2.0, host_credential_id=4, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data, connection_mode=proxy, target_cluster_status=RUNNING, latest_execution_start_time=2024-04-18T09:54:32.259+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=402}, listMap={add_region_ids=[], server_ids=[4, 5, 6, 7, 8, 9, 10, 11, 12, 13], add_idc_ids=[2, 3, 4], all_host_ids=[4, 5, 6, 7, 8, 9, 10, 11, 12, 13], add_host_ids=[4, 5, 6, 7, 8, 9, 10, 11, 12, 13], host_ids=[4, 5, 6, 7, 8, 9, 10, 11, 12, 13], zone_names=[zone4, zone1, zone3, zone2, zone5]}}, executor=10.191.69.58 2024-04-18 09:54:33.059 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2024-04-18 09:54:33.060 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-04-18 09:54:33.060 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-04-18 09:54:33.073 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:54:33.086 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:33.126 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:54:33.138 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:33.178 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.191.69.57 2024-04-18 09:54:33.205 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:33.207 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.191.69.57,result:SshResult(host=10.191.69.57, username=admin, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:33.245 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:54:33.257 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:33.297 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.191.69.57 2024-04-18 09:54:33.324 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:33.325 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.191.69.57,result:SshResult(host=10.191.69.57, username=admin, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:33.364 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:54:33.377 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:33.416 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.191.69.57 2024-04-18 09:54:33.441 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:33.442 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.191.69.57,result:SshResult(host=10.191.69.57, username=admin, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:33.443 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-04-18 09:54:33.481 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:54:33.493 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:33.533 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.191.69.57 2024-04-18 09:54:33.929 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:33.930 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.191.69.57,result:SshResult(host=10.191.69.57, username=admin, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:33.931 WARN 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-04-18 09:54:33.931 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-04-18 09:54:33.932 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-04-18 09:54:33.969 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:54:33.982 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:34.022 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.191.69.57 2024-04-18 09:54:34.418 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:34.419 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.191.69.57,result:SshResult(host=10.191.69.57, username=admin, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:34.420 WARN 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-04-18 09:54:34.421 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-04-18 09:54:34.422 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2024-04-18 09:54:34.422 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-04-18 09:54:34.423 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=11 2024-04-18 09:54:34.426 WARN 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=11 2024-04-18 09:54:34.427 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=11 2024-04-18 09:54:34.430 WARN 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=11 2024-04-18 09:54:34.431 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-04-18 09:54:34.432 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-04-18 09:54:34.458 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:54:34.471 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:34.511 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:54:34.523 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:34.563 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'netstat -ant| awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"'' on 10.191.69.57 2024-04-18 09:54:34.587 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'netstat -ant| awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"'' on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=sudo bash -c 'netstat -ant| awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:34.588 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'netstat -ant| awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"'' on 10.191.69.57,result:SshResult(host=10.191.69.57, username=admin, command=sudo bash -c 'netstat -ant| awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:34.627 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:54:34.639 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:34.679 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'netstat -ant| awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"'' on 10.191.69.57 2024-04-18 09:54:34.703 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'netstat -ant| awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"'' on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=sudo bash -c 'netstat -ant| awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:34.703 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'netstat -ant| awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"'' on 10.191.69.57,result:SshResult(host=10.191.69.57, username=admin, command=sudo bash -c 'netstat -ant| awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:34.716 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-04-18 09:54:34.742 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.191.69.57 2024-04-18 09:54:34.754 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-04-18 09:54:34.794 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' on 10.191.69.57 2024-04-18 09:54:34.819 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' on 10.191.69.57, result: SshResult(host=10.191.69.57, username=admin, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm', out=curl: (7) Failed connect to localhost:8080; Connection refused, err=, extOut=null, exitStatus=7) 2024-04-18 09:54:34.820 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' on 10.191.69.57,result:SshResult(host=10.191.69.57, username=admin, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm', out=curl: (7) Failed connect to localhost:8080; Connection refused, err=, extOut=null, exitStatus=7) 2024-04-18 09:54:34.821 INFO 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm', return code:7, output:curl: (7) Failed connect to localhost:8080; Connection refused, extOut:null 2024-04-18 09:54:34.821 ERROR 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm', exit code: 7, cause:null 2024-04-18 09:54:34.822 ERROR 22530 --- [pool-manual-subtask-executor13,0a1cccbb0e154590,6963f2a13ed8] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm' 'http://localhost:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.2.2-20240315150922.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:399) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:383) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:74) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:55) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:382) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:228) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:219) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:704) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$6315daad.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallO cpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:203) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:197) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:134) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Set state for subtask: 402, operation:RETRY, state: FAILED