2025-09-12 17:22:02.245 INFO 6621 --- [takeover-obs-process-check-executor2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-09-12 17:22:02.245 INFO 6621 --- [takeover-obs-process-check-executor3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-09-12 17:22:02.246 INFO 6621 --- [takeover-obs-process-check-executor2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-09-12 17:22:02.246 INFO 6621 --- [takeover-obs-process-check-executor1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-09-12 17:22:02.249 INFO 6621 --- [takeover-obs-process-check-executor1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-09-12 17:22:02.262 INFO 6621 --- [takeover-obs-process-check-executor3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.181:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-09-12 17:22:02.263 INFO 6621 --- [takeover-obs-process-check-executor2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.161:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-09-12 17:22:02.265 INFO 6621 --- [takeover-obs-process-check-executor1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.171:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-09-12 17:22:02.275 WARN 6621 --- [takeover-obs-process-check-executor3,,] c.o.o.o.i.c.ClusterOperationServiceImpl : doObsProcessCheck.createAgentExecutor failed! ObServer ip: 192.168.2.181 com.oceanbase.ocp.executor.exception.HttpRequestFailedException: [AgentClient]:http request is failed, response:Cannot get process info of can not found process observer listening on 2881, reason: %!v(MISSING) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.checkSuccess(HttpTemplate.java:479) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doPost(HttpTemplate.java:289) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.post(HttpTemplate.java:117) at com.oceanbase.ocp.executor.executor.AgentExecutor.getObProcessInfo(AgentExecutor.java:454) at com.oceanbase.ocp.obops.internal.cluster.ClusterOperationServiceImpl.doObsProcessCheck(ClusterOperationServiceImpl.java:1596) at com.oceanbase.ocp.obops.internal.cluster.ClusterOperationServiceImpl.lambda$obsProcessCheck$20(ClusterOperationServiceImpl.java:1552) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-09-12 17:22:02.330 ERROR 6621 --- [http-nio-0.0.0.0-8080-exec-2,ed709dd52646aa43,d575b1ea2279a768] c.o.o.s.c.trace.RequestTracingAspect : Unexpected API Error: [POST /api/v2/ob/clusters/takeOverPreCheck client=10.10.1.248, traceId=ed709dd52646aa43, args=TakeOverClusterPreCheckParam(connectionMode=proxy, clusterName=zhutang, vpcName=null, vpcId=1, obClusterId=null, rootSyspassword=xxx proxyropassword=xxx saveToCredential=true, address=192.168.2.171, port=2883),, duration=968 ms] java.lang.NullPointerException: null at com.oceanbase.ocp.obops.internal.cluster.ClusterOperationServiceImpl.obsProcessCheck(ClusterOperationServiceImpl.java:1582) at com.oceanbase.ocp.obops.internal.cluster.ClusterOperationServiceImpl.takeOverClusterPreCheck(ClusterOperationServiceImpl.java:1489) at com.oceanbase.ocp.obops.internal.cluster.ClusterOperationServiceImpl$$FastClassBySpringCGLIB$$d608ed50.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.cluster.ClusterOperationServiceImpl$$EnhancerBySpringCGLIB$$fc798af0.takeOverClusterPreCheck() at com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverClusterPreCheck(ObClusterController.java:178) at com.oceanbase.ocp.server.common.controller.ObClusterController$$FastClassBySpringCGLIB$$918db387.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:123) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.oceanbase.ocp.server.common.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:135) at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.server.common.controller.ObClusterController$$EnhancerBySpringCGLIB$$552ecb71.takeOverClusterPreCheck() at sun.reflect.GeneratedMethodAccessor1675.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:898) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:809) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:555) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:67) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.CredentialsExpiredFilter.doFilterInternal(CredentialsExpiredFilter.java:51) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter.doFilter(PreUsernamePasswordAuthenticationFilter.java:41) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:110) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter.doFilterInternal(TestLoginAuthenticationFilter.java:33) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationRequestFilter.doFilterInternal(Saml2WebSsoAuthenticationRequestFilter.java:192) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter.doFilterInternal(OAuth2AuthorizationRequestRedirectFilter.java:178) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:132) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at com.oceanbase.ocp.server.common.trace.TraceFilter.doFilter(TraceFilter.java:50) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:396) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:937) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.lang.Thread.run(Thread.java:750) 2025-09-12 17:22:02.330 ERROR 6621 --- [http-nio-0.0.0.0-8080-exec-2,ed709dd52646aa43,d575b1ea2279a768] c.o.o.s.c.e.RestExceptionHandler : Unhandled exception, uri=/api/v2/ob/clusters/takeOverPreCheck, duration=969 java.lang.NullPointerException: null at com.oceanbase.ocp.obops.internal.cluster.ClusterOperationServiceImpl.obsProcessCheck(ClusterOperationServiceImpl.java:1582) at com.oceanbase.ocp.obops.internal.cluster.ClusterOperationServiceImpl.takeOverClusterPreCheck(ClusterOperationServiceImpl.java:1489) at com.oceanbase.ocp.obops.internal.cluster.ClusterOperationServiceImpl$$FastClassBySpringCGLIB$$d608ed50.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.cluster.ClusterOperationServiceImpl$$EnhancerBySpringCGLIB$$fc798af0.takeOverClusterPreCheck() at com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverClusterPreCheck(ObClusterController.java:178) at com.oceanbase.ocp.server.common.controller.ObClusterController$$FastClassBySpringCGLIB$$918db387.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:123) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.oceanbase.ocp.server.common.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:135) at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.server.common.controller.ObClusterController$$EnhancerBySpringCGLIB$$552ecb71.takeOverClusterPreCheck() at sun.reflect.GeneratedMethodAccessor1675.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:898) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:809) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:555) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:67) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.CredentialsExpiredFilter.doFilterInternal(CredentialsExpiredFilter.java:51) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter.doFilter(PreUsernamePasswordAuthenticationFilter.java:41) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:110) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter.doFilterInternal(TestLoginAuthenticationFilter.java:33) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationRequestFilter.doFilterInternal(Saml2WebSsoAuthenticationRequestFilter.java:192) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter.doFilterInternal(OAuth2AuthorizationRequestRedirectFilter.java:178) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:132) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at com.oceanbase.ocp.server.common.trace.TraceFilter.doFilter(TraceFilter.java:50) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:396) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:937) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.lang.Thread.run(Thread.java:750) 2025-09-12 17:22:02.392 INFO 6621 --- [ocp-async-2,df2ce945bef5d44c,618efd0df2e93e55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:02.615 WARN 6621 --- [ocp-async-6,d7cccc1d2a2c6a02,7b2944678b058bc7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:04.244 INFO 6621 --- [prometheus-minute-schedule-1,e92991d837c25da0,fb8d5bb07c34d070] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1757668980, interval=60 2025-09-12 17:22:04.246 INFO 6621 --- [prometheus-minute-schedule-1,e92991d837c25da0,fb8d5bb07c34d070] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2025-09-12 17:22:04.392 INFO 6621 --- [ocp-async-6,ab032d8fed829740,2ba91b54760a5441] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:04.456 INFO 6621 --- [ocp-schedules-2,2d806272b92f5435,db25f1b35b0bd739] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-09-12 17:22:04.459 INFO 6621 --- [ocp-schedules-2,2d806272b92f5435,db25f1b35b0bd739] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-09-12 17:22:04.615 WARN 6621 --- [ocp-async-7,9cd28de592af364e,33e721ccb174cfce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:04.837 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.2.185:8080 refreshes policies... 2025-09-12 17:22:04.837 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-09-12 17:22:06.359 INFO 6621 --- [ocp-schedules-24,d69069b290b919bb,147625de2993fb08] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-09-12 17:22:06.385 WARN 6621 --- [scanner-1339557-1,d69069b290b919bb,4c0d171a911178c2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 6 2025-09-12 17:22:06.392 INFO 6621 --- [ocp-async-7,59e2ed9b5fbe31d8,5b6710880fa0ab34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:06.455 INFO 6621 --- [alarm-detect-16,d69069b290b919bb,acfde14b19c5280a] c.o.o.alarm.core.detect.MetricDetector : alarmType ocp_remote_server_time_diff_too_large detect time range 1757668890 ~ 1757668916 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricSnapshots={clock_diff_milliseconds=[SeriesMetricValues(metric=Metric{name='clock_diff_milliseconds', labels={app=HOST, host=192.168.2.161, host_name=localhost.localdomain, ob_cluster_id=0, ob_cluster_name=, svr_ip=192.168.2.161}}, data=[ValueData(timestamp=1757668905, value=2104612.0)])]}, metricEvaluateResults={clock_diff_milliseconds > 50=[ExpressionEvaluateResult(matchValue=50.0, metric=clock_diff_milliseconds, match=true, value=2104612.0, values={1757668905=2104612.0}, valueCount=1, labels={app=HOST, host=192.168.2.161, host_name=localhost.localdomain, ob_cluster_id=0, ob_cluster_name=, svr_ip=192.168.2.161})]}, matched=true, isContainValidValue=null, message=null, level=1) 2025-09-12 17:22:06.455 INFO 6621 --- [alarm-detect-16,d69069b290b919bb,acfde14b19c5280a] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ocp_remote_server_time_diff_too_large, target=alarm_template_id=0:host=192.168.2.161-1, currentState=Firing 2025-09-12 17:22:06.455 INFO 6621 --- [alarm-detect-16,d69069b290b919bb,acfde14b19c5280a] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ocp_remote_server_time_diff_too_large, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=192.168.2.161-1] 2025-09-12 17:22:06.476 INFO 6621 --- [alarm-detect-16,d69069b290b919bb,acfde14b19c5280a] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=3 2025-09-12 17:22:06.510 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-09-12 17:22:06.510 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-09-12 17:22:06.514 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-09-12 17:22:06.514 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : init spring 2025-09-12 17:22:06.514 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-09-12 17:22:06.514 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : starting 2025-09-12 17:22:06.517 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-09-12 17:22:06.517 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : init spring 2025-09-12 17:22:06.517 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-09-12 17:22:06.518 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : starting 2025-09-12 17:22:06.616 WARN 6621 --- [ocp-async-3,2483eca23c962661,c29307f98efea607] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:06.840 INFO 6621 --- [alarm-task-2,33c89928ce55b7c6,a5469c33d3b1e9ff] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-09-12T17:22:06.839968+08:00, lastProcessTime=2025-09-12T17:21:56.826678+08:00 2025-09-12 17:22:06.973 INFO 6621 --- [alarm-task-2,33c89928ce55b7c6,a5469c33d3b1e9ff] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-09-12T17:22:06.839968+08:00 2025-09-12 17:22:07.245 INFO 6621 --- [prometheus-minute-schedule-1,21eed7308f0a293a,dd2e985e80d730ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1757668980, interval=60 2025-09-12 17:22:07.249 INFO 6621 --- [prometheus-minute-schedule-1,21eed7308f0a293a,dd2e985e80d730ed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2025-09-12 17:22:07.426 INFO 6621 --- [ocp-schedules-29,5066810bccf1c34f,bb4355074693d59a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-09-12 17:22:07.520 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] o.s.cloud.commons.util.InetUtils : Cannot determine local hostname 2025-09-12 17:22:07.539 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-09-12 17:22:07.544 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-09-12 17:22:07.544 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-09-12 17:22:07.545 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-09-12 17:22:07.545 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-09-12 17:22:07.545 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-09-12 17:22:07.550 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-09-12 17:22:07.837 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.2.185:8080 refreshes policies... 2025-09-12 17:22:07.838 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-09-12 17:22:07.875 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-09-12 17:22:08.253 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.735461397S 2025-09-12 17:22:08.253 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-09-12 17:22:08.314 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-09-12 17:22:08.314 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-09-12 17:22:08.342 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-09-12 17:22:08.403 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-09-12 17:22:08.403 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-09-12 17:22:08.403 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-09-12 17:22:08.403 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-09-12 17:22:08.403 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-09-12 17:22:08.403 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-09-12 17:22:08.406 INFO 6621 --- [ocp-async-3,546394954cbdbc72,4da9f447ed3b96b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:08.412 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-09-12 17:22:08.544 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] o.s.boot.SpringApplication : Started application in 2.029 seconds (JVM running for 17832.333) 2025-09-12 17:22:08.544 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT2.030130537S 2025-09-12 17:22:08.550 INFO 6621 --- [ocp-schedules-1,b6eea7defaad4a80,bf17cd71ed4d96b4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-09-12 17:22:08.555 INFO 6621 --- [ocp-schedules-10,58a5a0d799001d46,74a7cccd23d1eb8f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[192.168.2.171, 192.168.2.161, 192.168.2.181] 2025-09-12 17:22:08.556 INFO 6621 --- [host-check-agent-status1,58a5a0d799001d46,ea4d572ce397fc30] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=192.168.2.171, originalStatus=AVAILABLE 2025-09-12 17:22:08.556 INFO 6621 --- [host-check-agent-status2,58a5a0d799001d46,dc6a851501c3f97a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=192.168.2.161, originalStatus=AVAILABLE 2025-09-12 17:22:08.557 INFO 6621 --- [host-check-agent-status3,58a5a0d799001d46,2e0464299289aaaa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=192.168.2.181, originalStatus=AVAILABLE 2025-09-12 17:22:08.557 INFO 6621 --- [ocp-schedules-26,e007ef58ac1c4da1,b9d6a0b2e99d57e1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[192.168.2.171, 192.168.2.161, 192.168.2.181] 2025-09-12 17:22:08.557 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=192.168.2.171, originalStatus=AVAILABLE 2025-09-12 17:22:08.558 INFO 6621 --- [ocp-schedules-15,c749c5b69ba8aa08,dcfdd4595f7f3e1e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[192.168.2.171, 192.168.2.161, 192.168.2.181] 2025-09-12 17:22:08.559 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=192.168.2.161, originalStatus=AVAILABLE 2025-09-12 17:22:08.560 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=192.168.2.181, originalStatus=AVAILABLE 2025-09-12 17:22:08.561 INFO 6621 --- [host-check-agent-status2,58a5a0d799001d46,dc6a851501c3f97a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-09-12 17:22:08.562 INFO 6621 --- [host-check-agent-status3,58a5a0d799001d46,2e0464299289aaaa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-09-12 17:22:08.562 INFO 6621 --- [host-check-agent-status1,58a5a0d799001d46,ea4d572ce397fc30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-09-12 17:22:08.563 INFO 6621 --- [ocp-schedules-19,ae31d821a3246691,90c76b01915456de] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-09-12 17:22:08.563 INFO 6621 --- [host-check-agent-status2,58a5a0d799001d46,dc6a851501c3f97a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-09-12 17:22:08.563 INFO 6621 --- [ocp-schedules-3,c867d400d42fddd9,d8770210660b3334] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-09-12 17:22:08.563 INFO 6621 --- [host-check-agent-status1,58a5a0d799001d46,ea4d572ce397fc30] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-09-12 17:22:08.564 INFO 6621 --- [host-check-agent-status3,58a5a0d799001d46,2e0464299289aaaa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-09-12 17:22:08.566 INFO 6621 --- [ocp-schedules-3,c867d400d42fddd9,d8770210660b3334] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-09-12 17:22:08.566 INFO 6621 --- [ocp-schedules-4,caed16b705bbcabc,0c20fe8aa0c6d420] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-09-12 17:22:08.567 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-09-12 17:22:08.567 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-09-12 17:22:08.567 INFO 6621 --- [ocp-schedules-8,b5c499be1f7ee5b7,8041b11b05a19533] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-09-12 17:22:08.567 INFO 6621 --- [ocp-schedules-19,ae31d821a3246691,90c76b01915456de] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-09-12 17:22:08.568 INFO 6621 --- [ocp-schedules-29,f024926d02e8cf80,5d435d5315a079dd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-09-12 17:22:08.569 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-09-12 17:22:08.569 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-09-12 17:22:08.570 INFO 6621 --- [ocp-schedules-29,f024926d02e8cf80,5d435d5315a079dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-09-12 17:22:08.570 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-09-12 17:22:08.571 INFO 6621 --- [host-check-clockdiff2,c749c5b69ba8aa08,3d949c0a958b2760] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.2.161], timeoutMillis=20000 2025-09-12 17:22:08.572 INFO 6621 --- [ocp-schedules-29,f024926d02e8cf80,5d435d5315a079dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2025-09-12 17:22:08.573 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-09-12 17:22:08.574 INFO 6621 --- [ocp-schedules-18,d4e259bdeb1be6be,df199688d7936923] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-09-12 17:22:08.574 INFO 6621 --- [ocp-schedules-8,b5c499be1f7ee5b7,8041b11b05a19533] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-09-12 17:22:08.577 INFO 6621 --- [ocp-schedules-17,2f8c5b6b4850aeb2,a2968c46e161b531] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-09-12 17:22:08.579 INFO 6621 --- [host-check-clockdiff1,c749c5b69ba8aa08,06a69f4f6e0dd18e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.2.171], timeoutMillis=20000 2025-09-12 17:22:08.583 INFO 6621 --- [ocp-schedules-18,d4e259bdeb1be6be,df199688d7936923] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-09-12 17:22:08.589 INFO 6621 --- [ocp-schedules-24,0e83863ee371fa59,af02899c8b0a4d98] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-09-12 17:22:08.589 INFO 6621 --- [ocp-schedules-24,0e83863ee371fa59,af02899c8b0a4d98] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-09-12 17:22:08.589 INFO 6621 --- [host-check-agent-status2,58a5a0d799001d46,dc6a851501c3f97a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.161:62888/api/v1/agent/status, params:null 2025-09-12 17:22:08.589 INFO 6621 --- [ocp-schedules-17,2f8c5b6b4850aeb2,a2968c46e161b531] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-09-12 17:22:08.590 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.171:62888/api/v1/time, params:null 2025-09-12 17:22:08.592 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=192.168.2.171, timestamp=2025-09-12T16:47:03.980023889+08:00 2025-09-12 17:22:08.592 INFO 6621 --- [host-check-agent-status3,58a5a0d799001d46,2e0464299289aaaa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.181:62888/api/v1/agent/status, params:null 2025-09-12 17:22:08.593 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-09-12T17:22:08.592+08:00 2025-09-12 17:22:08.594 INFO 6621 --- [host-check-agent-status1,58a5a0d799001d46,ea4d572ce397fc30] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.171:62888/api/v1/agent/status, params:null 2025-09-12 17:22:08.596 INFO 6621 --- [host-check-agent-status2,58a5a0d799001d46,dc6a851501c3f97a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=192.168.2.161 2025-09-12 17:22:08.596 INFO 6621 --- [host-check-agent-status2,58a5a0d799001d46,dc6a851501c3f97a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=192.168.2.161 2025-09-12 17:22:08.596 INFO 6621 --- [host-check-agent-status2,58a5a0d799001d46,dc6a851501c3f97a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=192.168.2.161 2025-09-12 17:22:08.596 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.181:62888/api/v1/time, params:null 2025-09-12 17:22:08.597 INFO 6621 --- [host-check-agent-status1,58a5a0d799001d46,ea4d572ce397fc30] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=192.168.2.171 2025-09-12 17:22:08.597 INFO 6621 --- [host-check-agent-status1,58a5a0d799001d46,ea4d572ce397fc30] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=192.168.2.171 2025-09-12 17:22:08.597 INFO 6621 --- [host-check-agent-status1,58a5a0d799001d46,ea4d572ce397fc30] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=192.168.2.171 2025-09-12 17:22:08.598 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-09-12T17:22:08.592+08:00 and status AVAILABLE 2025-09-12 17:22:08.598 INFO 6621 --- [host-check-agent-status3,58a5a0d799001d46,2e0464299289aaaa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=192.168.2.181 2025-09-12 17:22:08.598 INFO 6621 --- [host-check-agent-status3,58a5a0d799001d46,2e0464299289aaaa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=192.168.2.181 2025-09-12 17:22:08.598 INFO 6621 --- [host-check-agent-status3,58a5a0d799001d46,2e0464299289aaaa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=192.168.2.181 2025-09-12 17:22:08.600 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-09-12 17:22:08.605 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.161:62888/api/v1/time, params:null 2025-09-12 17:22:08.606 INFO 6621 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ztocp, obClusterId=1757648497, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-09-12T11:52:54+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/data/1","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/data/log1","SqlPort":2881,"SvrPort":2882}, tenantCount=0) 2025-09-12 17:22:08.606 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-09-12 17:22:08.606 INFO 6621 --- [ocp-schedules-3,c867d400d42fddd9,d8770210660b3334] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-09-12 17:22:08.606 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=192.168.2.161, timestamp=2025-09-12T16:47:03.994807690+08:00 2025-09-12 17:22:08.607 INFO 6621 --- [ocp-schedules-19,ae31d821a3246691,90c76b01915456de] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-09-12 17:22:08.607 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-09-12T17:22:08.606+08:00 2025-09-12 17:22:08.607 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-09-12 17:22:08.609 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-09-12T17:22:08.606+08:00 and status AVAILABLE 2025-09-12 17:22:08.611 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-09-12 17:22:08.618 WARN 6621 --- [ocp-async-1,962d54794994b993,5fe7e5067be62f4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:08.622 INFO 6621 --- [pool-overview-container-3,2c2b34eef7a0a984,e520e84b951a0ac7] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-09-12 17:22:08.623 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-09-12 17:22:08.624 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-09-12 17:22:08.625 INFO 6621 --- [host-check-clockdiff2,c749c5b69ba8aa08,3d949c0a958b2760] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1757668928 -2104612 -2104612 , error= 2025-09-12 17:22:08.627 INFO 6621 --- [host-check-clockdiff3,c749c5b69ba8aa08,f4c4c4846c8ab7a6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.2.181], timeoutMillis=20000 2025-09-12 17:22:08.630 INFO 6621 --- [host-check-clockdiff1,c749c5b69ba8aa08,06a69f4f6e0dd18e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1757668928 -2104612 -2104612 , error= 2025-09-12 17:22:08.632 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=192.168.2.181, timestamp=2025-09-12T16:47:04.018383718+08:00 2025-09-12 17:22:08.632 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-09-12T17:22:08.632+08:00 2025-09-12 17:22:08.634 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-09-12T17:22:08.632+08:00 and status AVAILABLE 2025-09-12 17:22:08.636 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-09-12 17:22:08.637 INFO 6621 --- [pool-12-thread-16,2c2b34eef7a0a984,b342c99dc08e16f0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-09-12 17:22:08.637 INFO 6621 --- [pool-12-thread-16,2c2b34eef7a0a984,b342c99dc08e16f0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-09-12 17:22:08.644 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.171:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-09-12 17:22:08.649 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-09-12 17:22:08.650 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-09-12 17:22:08.650 INFO 6621 --- [pool-12-thread-16,2c2b34eef7a0a984,b342c99dc08e16f0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-09-12 17:22:08.652 INFO 6621 --- [pool-12-thread-16,2c2b34eef7a0a984,b342c99dc08e16f0] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-09-12 17:22:08.652 INFO 6621 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-09-12 17:22:08.652 INFO 6621 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ztocp, obClusterId=1757648497, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-09-12T11:52:54+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/data/1","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/data/log1","SqlPort":2881,"SvrPort":2882}, tenantCount=0) 2025-09-12 17:22:08.652 INFO 6621 --- [ocp-schedules-8,b5c499be1f7ee5b7,8041b11b05a19533] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-09-12 17:22:08.654 INFO 6621 --- [host-check-clockdiff3,c749c5b69ba8aa08,f4c4c4846c8ab7a6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1757668928 -2104612 -2104612 , error= 2025-09-12 17:22:08.657 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.161:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-09-12 17:22:08.664 INFO 6621 --- [pool-12-thread-14,2c2b34eef7a0a984,603564bdb91a2440] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-09-12 17:22:08.664 INFO 6621 --- [pool-12-thread-13,2c2b34eef7a0a984,517e8eb271c4e0ea] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-09-12 17:22:08.665 INFO 6621 --- [pool-12-thread-15,2c2b34eef7a0a984,070ef788fec3392b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-09-12 17:22:08.667 INFO 6621 --- [pool-12-thread-13,2c2b34eef7a0a984,517e8eb271c4e0ea] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-09-12 17:22:08.667 INFO 6621 --- [pool-12-thread-13,2c2b34eef7a0a984,517e8eb271c4e0ea] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-09-12 17:22:08.669 INFO 6621 --- [ForkJoinPool-7-worker-10,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-09-12 17:22:08.670 INFO 6621 --- [ForkJoinPool-7-worker-10,2c2b34eef7a0a984,e3bae7e283f7f366] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of ztocp 2025-09-12 17:22:08.670 INFO 6621 --- [ForkJoinPool-7-worker-10,2c2b34eef7a0a984,e3bae7e283f7f366] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=ztocp, obClusterId=1757648497, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2025-09-12 17:22:08.670 INFO 6621 --- [ForkJoinPool-7-worker-10,2c2b34eef7a0a984,e3bae7e283f7f366] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of ztocp 2025-09-12 17:22:08.670 INFO 6621 --- [pool-12-thread-14,2c2b34eef7a0a984,603564bdb91a2440] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-09-12 17:22:08.672 INFO 6621 --- [ForkJoinPool-7-worker-3,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-09-12 17:22:08.673 INFO 6621 --- [ForkJoinPool-7-worker-3,2c2b34eef7a0a984,0261ca1230e26bf1] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=ztocp, obClusterId=1757648497, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2025-09-12 17:22:08.673 INFO 6621 --- [pool-12-thread-15,2c2b34eef7a0a984,070ef788fec3392b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-09-12 17:22:08.675 INFO 6621 --- [pool-overview-container-2,0929956fbd477924,2ceb2b12bd9a5f3c] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-09-12 17:22:08.679 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.181:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-09-12 17:22:08.684 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-09-12 17:22:08.693 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-09-12 17:22:08.694 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-09-12 17:22:08.704 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-09-12 17:22:08.716 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.171:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-09-12 17:22:08.721 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-09-12 17:22:08.725 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.161:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-09-12 17:22:08.733 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-09-12 17:22:08.746 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.181:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-09-12 17:22:08.761 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-09-12 17:22:08.769 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-09-12 17:22:08.769 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-09-12 17:22:08.776 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-09-12 17:22:08.788 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.171:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-09-12 17:22:08.788 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-09-12 17:22:08.792 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.161:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-09-12 17:22:08.796 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-09-12 17:22:08.810 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.181:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-09-12 17:22:08.816 INFO 6621 --- [ocp-schedules-20,7020c037d0d475f3,9dbf3a330b7a0643] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-09-12 17:22:08.816 INFO 6621 --- [ocp-schedules-20,7020c037d0d475f3,9dbf3a330b7a0643] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-09-12 17:22:08.828 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-09-12 17:22:08.834 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-09-12 17:22:08.838 INFO 6621 --- [host-check-host-status1,e007ef58ac1c4da1,102e3887afa8faf1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-09-12 17:22:08.842 INFO 6621 --- [host-check-host-status2,e007ef58ac1c4da1,6ada86c12426c456] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-09-12 17:22:08.847 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-09-12 17:22:08.854 INFO 6621 --- [host-check-host-status3,e007ef58ac1c4da1,f8f9a37519b0ebba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-09-12 17:22:08.890 INFO 6621 --- [ocp-schedules-9,74f61311fb833b31,402f493fac15ef09] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-09-12 17:22:09.137 INFO 6621 --- [ocp-async-8,45e358dd03eb14c8,0681745b8510ee3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-09-12 17:22:09.342 INFO 6621 --- [ocp-schedules-30,99987587272aa0a3,752a2e4b851fea90] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-09-12 17:22:09.343 INFO 6621 --- [ocp-schedules-30,99987587272aa0a3,752a2e4b851fea90] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-09-12 17:22:09.343 INFO 6621 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-09-12 17:22:09.343 INFO 6621 --- [ocp-schedules-30,99987587272aa0a3,752a2e4b851fea90] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-09-12 17:22:09.481 INFO 6621 --- [ocp-schedules-6,4bf43645ede04feb,252cdb1e5d41e044] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-09-12 17:22:09.488 INFO 6621 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-09-12 17:22:09.644 INFO 6621 --- [ocp-schedules-13,1f6b7d1bd44b9c71,cfbdbd7e4cb3d5d2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-09-12 17:22:09.646 INFO 6621 --- [ocp-schedules-13,1f6b7d1bd44b9c71,cfbdbd7e4cb3d5d2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-09-12 17:22:09.646 INFO 6621 --- [ocp-schedules-13,1f6b7d1bd44b9c71,cfbdbd7e4cb3d5d2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-09-12 17:22:10.005 INFO 6621 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-09-12 17:22:10.007 INFO 6621 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-09-12 17:22:10.009 INFO 6621 --- [ocp-schedules-28,54ff9ef63e5be96e,4b7a87aff5de75c3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-09-12 17:22:10.009 INFO 6621 --- [ocp-schedules-28,54ff9ef63e5be96e,4b7a87aff5de75c3] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-09-12 17:22:10.012 INFO 6621 --- [ocp-schedules-4,b71b1ccc82cebe7d,eeaf48fe9c753ab5] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-09-12 17:22:10.012 INFO 6621 --- [ocp-schedules-4,b71b1ccc82cebe7d,eeaf48fe9c753ab5] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-09-12 17:22:10.018 INFO 6621 --- [pool-83-thread-1,54ff9ef63e5be96e,22ad2fba39c2db00] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-09-12 17:22:10.027 WARN 6621 --- [ocp-schedules-28,54ff9ef63e5be96e,4b7a87aff5de75c3] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ztocp,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ztocp,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:167) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:144) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$7f4debd5.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:135) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$dd406329.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-09-12 17:22:10.027 INFO 6621 --- [ocp-schedules-28,54ff9ef63e5be96e,4b7a87aff5de75c3] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-09-12 17:22:10.046 INFO 6621 --- [ocp-schedules-2,a776016e928954c3,db54b903ae605759] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-09-12 17:22:10.046 INFO 6621 --- [ocp-schedules-2,a776016e928954c3,db54b903ae605759] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-09-12 17:22:10.244 INFO 6621 --- [prometheus-minute-schedule-1,1a9c9a29be10dc6e,694c97ee0a8332ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1757668980, interval=60 2025-09-12 17:22:10.246 INFO 6621 --- [prometheus-minute-schedule-1,1a9c9a29be10dc6e,694c97ee0a8332ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2025-09-12 17:22:10.395 INFO 6621 --- [ocp-schedules-16,d97fa41df6ed67be,42c19202839babe0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-09-12 17:22:10.395 INFO 6621 --- [ocp-async-4,4ab952122a06c501,eb71c24fca033645] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:10.508 INFO 6621 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 7, ruleSize = 7. 2025-09-12 17:22:10.619 WARN 6621 --- [ocp-async-10,76b12b842195cb45,0d27d4dc8e6cdcc7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:10.625 INFO 6621 --- [ocp-schedules-3,1e7c18b035eb3f6f,3e1bc35ad2d46464] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-09-12 17:22:10.705 INFO 6621 --- [ocp-schedules-22,3abed1c99aa733e0,e9cbd25696fa4cfd] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-09-12 17:22:10.705 INFO 6621 --- [ocp-schedules-22,3abed1c99aa733e0,e9cbd25696fa4cfd] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=822994477505384448 2025-09-12 17:22:10.712 INFO 6621 --- [ocp-schedules-22,3abed1c99aa733e0,e9cbd25696fa4cfd] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-09-12 17:22:10.712 INFO 6621 --- [ocp-schedules-22,3abed1c99aa733e0,e9cbd25696fa4cfd] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=823034145949159424 2025-09-12 17:22:10.715 INFO 6621 --- [ocp-schedules-22,3abed1c99aa733e0,e9cbd25696fa4cfd] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-09-12 17:22:10.715 INFO 6621 --- [ocp-schedules-22,3abed1c99aa733e0,e9cbd25696fa4cfd] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=822990953795096576 2025-09-12 17:22:10.839 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.2.185:8080 refreshes policies... 2025-09-12 17:22:10.840 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-09-12 17:22:11.034 INFO 6621 --- [ocp-schedules-13,eae781243f128aeb,c559678b939a01dd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-09-12 17:22:11.856 INFO 6621 --- [ocp-schedules-10,bdbdb6c16a971361,0443c11961ebb578] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-09-12 17:22:11.856 INFO 6621 --- [ocp-schedules-10,bdbdb6c16a971361,0443c11961ebb578] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-09-12 17:22:11.864 INFO 6621 --- [ocp-schedules-10,bdbdb6c16a971361,0443c11961ebb578] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-09-12 17:22:11.864 INFO 6621 --- [ocp-schedules-10,bdbdb6c16a971361,0443c11961ebb578] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-09-12 17:22:12.033 INFO 6621 --- [ocp-schedules-16,aa01ae3dc108fb97,c7eb020e837412e7] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-09-12 17:22:12.211 INFO 6621 --- [ocp-schedules-22,13b6a84e7246262c,69640786e15b0e5c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-09-12T17:22:12.211+08:00, clusters=[1] 2025-09-12 17:22:12.212 INFO 6621 --- [ocp-schedules-22,13b6a84e7246262c,69640786e15b0e5c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ztocp, trans stat result is empty. 2025-09-12 17:22:12.396 INFO 6621 --- [ocp-async-10,81cd98b7003151a0,a0e31e5932a7e94e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:12.405 INFO 6621 --- [ocp-schedules-23,2bb3c782a0c58f6a,cc83e62ab2be46f1] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-09-12 17:22:12.414 INFO 6621 --- [ocp-schedules-23,2bb3c782a0c58f6a,cc83e62ab2be46f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-09-12 17:22:12.415 WARN 6621 --- [ocp-schedules-23,2bb3c782a0c58f6a,cc83e62ab2be46f1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-09-12 17:22:12.416 INFO 6621 --- [ocp-schedules-23,2bb3c782a0c58f6a,cc83e62ab2be46f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-09-12 17:22:12.417 WARN 6621 --- [ocp-schedules-23,2bb3c782a0c58f6a,cc83e62ab2be46f1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-09-12 17:22:12.427 INFO 6621 --- [ocp-schedules-23,2bb3c782a0c58f6a,cc83e62ab2be46f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-09-12 17:22:12.428 INFO 6621 --- [ocp-schedules-23,2bb3c782a0c58f6a,cc83e62ab2be46f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-09-12 17:22:12.445 INFO 6621 --- [ocp-schedules-23,2bb3c782a0c58f6a,cc83e62ab2be46f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-09-12 17:22:12.446 INFO 6621 --- [ocp-schedules-23,2bb3c782a0c58f6a,cc83e62ab2be46f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-09-12 17:22:12.461 INFO 6621 --- [ocp-schedules-23,2bb3c782a0c58f6a,cc83e62ab2be46f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-09-12 17:22:12.462 INFO 6621 --- [ocp-schedules-23,2bb3c782a0c58f6a,cc83e62ab2be46f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-09-12 17:22:12.479 INFO 6621 --- [ocp-schedules-23,2bb3c782a0c58f6a,cc83e62ab2be46f1] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=4 2025-09-12 17:22:12.571 INFO 6621 --- [ocp-schedules-26,f95ec6fa7c1aaa5a,26b6bf97e17b8718] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-09-12T17:22:12.571+08:00, clusters=[1] 2025-09-12 17:22:12.574 INFO 6621 --- [ocp-schedules-26,f95ec6fa7c1aaa5a,26b6bf97e17b8718] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ztocp for slowSqlStat, sql count is 0. 2025-09-12 17:22:12.619 WARN 6621 --- [ocp-async-6,aac83a9c2c03e9eb,de83492039aa4c55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:13.244 INFO 6621 --- [prometheus-minute-schedule-1,ab437bc7794864de,ebfdc687ba1a51b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1757668980, interval=60 2025-09-12 17:22:13.247 INFO 6621 --- [prometheus-minute-schedule-1,ab437bc7794864de,ebfdc687ba1a51b7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2025-09-12 17:22:13.840 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.2.185:8080 refreshes policies... 2025-09-12 17:22:13.841 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-09-12 17:22:14.396 INFO 6621 --- [ocp-async-6,871e850ba02a0d2e,7b57104ea1b77141] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:14.464 INFO 6621 --- [ocp-schedules-27,a0421068b2b69034,b7131563107f1fce] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-09-12 17:22:14.466 INFO 6621 --- [ocp-schedules-27,a0421068b2b69034,b7131563107f1fce] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-09-12 17:22:14.619 WARN 6621 --- [ocp-async-1,06dbb4fd803be52c,a8144beacb534966] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:14.752 INFO 6621 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1757666760, nodeCount=171, affectRows=171 2025-09-12 17:22:14.753 INFO 6621 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1757666760, nodeCount=170, validCount=170 2025-09-12 17:22:14.754 ERROR 6621 --- [metric-archive-second-2,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=1182826624) Duplicate entry '822990464344985600-1757666700' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=1182826624) Duplicate entry '822990464344985600-1757666700' for key 'PRIMARY' 2025-09-12 17:22:14.757 INFO 6621 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1757666760, nodeCount=172, affectRows=172 2025-09-12 17:22:14.758 INFO 6621 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1757666760, nodeCount=172, validCount=172 2025-09-12 17:22:15.104 INFO 6621 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1757668860, nodeCount=1, affectRows=1 2025-09-12 17:22:15.105 INFO 6621 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1757668860, nodeCount=2, affectRows=2 2025-09-12 17:22:15.191 INFO 6621 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1757668860, nodeCount=1000, affectRows=1000 2025-09-12 17:22:15.194 INFO 6621 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1757668860, nodeCount=995, validCount=759 2025-09-12 17:22:15.714 INFO 6621 --- [prometheus-second-schedule-1,1fed22aed6af3708,0b02729e05ab45c0] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=3, collectTaskCount=3 2025-09-12 17:22:16.244 INFO 6621 --- [prometheus-minute-schedule-1,1b93ac2d914dd4e1,b926ab43a6011e03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1757668980, interval=60 2025-09-12 17:22:16.246 INFO 6621 --- [prometheus-minute-schedule-1,1b93ac2d914dd4e1,b926ab43a6011e03] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2025-09-12 17:22:16.396 INFO 6621 --- [ocp-async-1,d369f7c0662cbc71,02089768bf9f2edb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:16.493 INFO 6621 --- [ocp-schedules-22,68d6f5e4872cc8a9,57a976c6abfb6ceb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-09-12 17:22:16.512 WARN 6621 --- [scanner-1340098-1,68d6f5e4872cc8a9,0bec5c6c834916d3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 6 2025-09-12 17:22:16.619 WARN 6621 --- [ocp-async-9,7d2171f00d290ab8,4c57554648e53530] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:16.826 INFO 6621 --- [alarm-task-2,33c89928ce55b7c6,a5469c33d3b1e9ff] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-09-12T17:22:16.825088+08:00, lastProcessTime=2025-09-12T17:22:06.839968+08:00 2025-09-12 17:22:16.841 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.2.185:8080 refreshes policies... 2025-09-12 17:22:16.842 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-09-12 17:22:16.842 INFO 6621 --- [ocp-schedules-29,1d667674506ba69e,93ceaf75ad797ea5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-09-12 17:22:16.845 INFO 6621 --- [ocp-schedules-29,1d667674506ba69e,93ceaf75ad797ea5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[bd9d56f1-2d09-45eb-984f-2776b048c82b] 2025-09-12 17:22:16.845 INFO 6621 --- [ocp-schedules-29,1d667674506ba69e,93ceaf75ad797ea5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[bd9d56f1-2d09-45eb-984f-2776b048c82b] 2025-09-12 17:22:16.950 INFO 6621 --- [alarm-task-2,33c89928ce55b7c6,a5469c33d3b1e9ff] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-09-12T17:22:16.825088+08:00 2025-09-12 17:22:17.430 INFO 6621 --- [ocp-schedules-28,e56fe25ca1572964,6c2805f0ea500104] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-09-12 17:22:18.184 INFO 6621 --- [ocp-schedules-4,459ffec81343a4cb,a63c5fb2d2c30a8a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-09-12 17:22:18.185 INFO 6621 --- [ocp-schedules-4,459ffec81343a4cb,a63c5fb2d2c30a8a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-09-12 17:22:18.199 INFO 6621 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-09-12 17:22:18.200 INFO 6621 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-09-12 17:22:18.205 INFO 6621 --- [ocp-schedules-4,459ffec81343a4cb,a63c5fb2d2c30a8a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-09-12 17:22:18.396 INFO 6621 --- [ocp-async-9,f10eb1131304aa17,d51509b6bd0814f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:18.619 WARN 6621 --- [ocp-async-2,12ee4851896fc6e1,971443208322ccf2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:19.137 INFO 6621 --- [ocp-async-10,e9903eaa2f074595,d169e336a96d8195] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-09-12 17:22:19.244 INFO 6621 --- [prometheus-minute-schedule-1,9fd5e377e4ab3905,d027ecde0b278d16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1757668980, interval=60 2025-09-12 17:22:19.246 INFO 6621 --- [prometheus-minute-schedule-1,9fd5e377e4ab3905,d027ecde0b278d16] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2025-09-12 17:22:19.466 INFO 6621 --- [ocp-schedules-4,ee3d80bacec0a378,e702dc359685aab0] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-09-12 17:22:19.468 INFO 6621 --- [ocp-schedules-4,ee3d80bacec0a378,e702dc359685aab0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-09-12 17:22:19.493 ERROR 6621 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ztocp, sys] 2025-09-12 17:22:19.495 ERROR 6621 --- [ocp-schedules-4,ee3d80bacec0a378,e702dc359685aab0] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ztocp,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:81) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor926.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ztocp,sys at sun.reflect.GeneratedConstructorAccessor405.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$daf268a.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:73) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-09-12 17:22:19.495 INFO 6621 --- [ocp-schedules-4,ee3d80bacec0a378,e702dc359685aab0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-09-12 17:22:19.842 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.2.185:8080 refreshes policies... 2025-09-12 17:22:19.843 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-09-12 17:22:20.396 INFO 6621 --- [ocp-async-8,d6b36e25639ef497,ee02d262cd817510] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:20.619 WARN 6621 --- [ocp-async-4,bd78b80253baa742,be11493f0dd2bdc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:22.244 INFO 6621 --- [prometheus-minute-schedule-1,037e38e643d92a92,168da08443ef2b17] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1757668980, interval=60 2025-09-12 17:22:22.247 INFO 6621 --- [prometheus-minute-schedule-1,037e38e643d92a92,168da08443ef2b17] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2025-09-12 17:22:22.396 INFO 6621 --- [ocp-async-4,ada0d54aab417e35,9650af4d87812be1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:22.403 INFO 6621 --- [ocp-schedules-13,022d974096651300,030d7997169dcc92] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.2.185, 192.168.2.171, 192.168.2.161, 192.168.2.181] 2025-09-12 17:22:22.403 INFO 6621 --- [ocp-schedules-13,022d974096651300,030d7997169dcc92] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-09-12 17:22:22.403 INFO 6621 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-09-12 17:22:22.403 INFO 6621 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-09-12 17:22:22.403 INFO 6621 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-09-12 17:22:22.408 INFO 6621 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-09-12 17:22:22.409 INFO 6621 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-09-12 17:22:22.409 INFO 6621 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-09-12 17:22:22.409 INFO 6621 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-09-12 17:22:22.410 INFO 6621 --- [ocp-schedules-13,022d974096651300,030d7997169dcc92] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-09-12 17:22:22.410 WARN 6621 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-09-12 17:22:22.411 INFO 6621 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-09-12 17:22:22.411 INFO 6621 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-09-12 17:22:22.411 INFO 6621 --- [ocp-schedules-13,022d974096651300,030d7997169dcc92] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-09-12 17:22:22.412 WARN 6621 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-09-12 17:22:22.416 INFO 6621 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.185:62889/api/v1/log/alarms, params:null 2025-09-12 17:22:22.423 INFO 6621 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.181:62889/api/v1/log/alarms, params:null 2025-09-12 17:22:22.426 INFO 6621 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.171:62889/api/v1/log/alarms, params:null 2025-09-12 17:22:22.439 INFO 6621 --- [ocp-schedules-13,022d974096651300,030d7997169dcc92] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.161:62889/api/v1/log/alarms, params:null 2025-09-12 17:22:22.442 WARN 6621 --- [ocp-schedules-13,022d974096651300,030d7997169dcc92] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-09-12 17:22:22.446 ERROR 6621 --- [ocp-schedules-13,022d974096651300,030d7997169dcc92] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor955.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1130) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1eb602fd.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.2.185:62889 [/192.168.2.185] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.2.185:62889 [/192.168.2.185] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2025-09-12 17:22:22.619 WARN 6621 --- [ocp-async-10,99a00ae594f7f019,1b1846ee0cf9fbbb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:22.843 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.2.185:8080 refreshes policies... 2025-09-12 17:22:22.844 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-09-12 17:22:22.866 INFO 6621 --- [ocp-updater-0,,] c.o.ocp.obsdk.connector.ObVersionHolder : Invalid expired ob version cache 2025-09-12 17:22:24.396 INFO 6621 --- [ocp-async-10,297e16524d88fe8b,a0ca30ab9fc5bd98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:24.469 INFO 6621 --- [ocp-schedules-7,22af2917a443636e,0e82b51bbfc7a631] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-09-12 17:22:24.471 INFO 6621 --- [ocp-schedules-7,22af2917a443636e,0e82b51bbfc7a631] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-09-12 17:22:24.620 WARN 6621 --- [ocp-async-6,43f0abad06134a71,679efa721a56c4ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:25.244 INFO 6621 --- [prometheus-minute-schedule-1,bd702dcd497d8ca5,55992e0f81bfa3ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1757668980, interval=60 2025-09-12 17:22:25.248 INFO 6621 --- [prometheus-minute-schedule-1,bd702dcd497d8ca5,55992e0f81bfa3ae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2025-09-12 17:22:25.844 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.2.185:8080 refreshes policies... 2025-09-12 17:22:25.844 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-09-12 17:22:26.396 INFO 6621 --- [ocp-async-6,49dd1340d2245f55,2b8adfc240bbea9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:26.566 INFO 6621 --- [ocp-schedules-2,19befb8ddd3b8207,477ed63f68f9e375] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-09-12 17:22:26.580 WARN 6621 --- [scanner-1340205-1,19befb8ddd3b8207,b10199571e7beada] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 6 2025-09-12 17:22:26.620 WARN 6621 --- [ocp-async-1,cd89ba27cbd0fe2b,efff54416794c7ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:26.659 INFO 6621 --- [alarm-detect-18,19befb8ddd3b8207,49ce91372098dd25] c.o.o.alarm.core.detect.MetricDetector : alarmType ocp_remote_server_time_diff_too_large detect time range 1757668906 ~ 1757668936 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricSnapshots={clock_diff_milliseconds=[SeriesMetricValues(metric=Metric{name='clock_diff_milliseconds', labels={app=HOST, host=192.168.2.161, host_name=localhost.localdomain, ob_cluster_id=0, ob_cluster_name=, svr_ip=192.168.2.161}}, data=[ValueData(timestamp=1757668920, value=2104612.0)])]}, metricEvaluateResults={clock_diff_milliseconds > 50=[ExpressionEvaluateResult(matchValue=50.0, metric=clock_diff_milliseconds, match=true, value=2104612.0, values={1757668920=2104612.0}, valueCount=1, labels={app=HOST, host=192.168.2.161, host_name=localhost.localdomain, ob_cluster_id=0, ob_cluster_name=, svr_ip=192.168.2.161})]}, matched=true, isContainValidValue=null, message=null, level=1) 2025-09-12 17:22:26.659 INFO 6621 --- [alarm-detect-18,19befb8ddd3b8207,49ce91372098dd25] c.o.o.alarm.core.detect.MetricDetector : alarmType ocp_remote_server_time_diff_too_large detect time range 1757668906 ~ 1757668936 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricSnapshots={clock_diff_milliseconds=[SeriesMetricValues(metric=Metric{name='clock_diff_milliseconds', labels={app=HOST, host=192.168.2.171, host_name=localhost.localdomain, ob_cluster_id=0, ob_cluster_name=, svr_ip=192.168.2.171}}, data=[ValueData(timestamp=1757668920, value=2104612.0)])]}, metricEvaluateResults={clock_diff_milliseconds > 50=[ExpressionEvaluateResult(matchValue=50.0, metric=clock_diff_milliseconds, match=true, value=2104612.0, values={1757668920=2104612.0}, valueCount=1, labels={app=HOST, host=192.168.2.171, host_name=localhost.localdomain, ob_cluster_id=0, ob_cluster_name=, svr_ip=192.168.2.171})]}, matched=true, isContainValidValue=null, message=null, level=1) 2025-09-12 17:22:26.659 INFO 6621 --- [alarm-detect-18,19befb8ddd3b8207,49ce91372098dd25] c.o.o.alarm.core.detect.MetricDetector : alarmType ocp_remote_server_time_diff_too_large detect time range 1757668906 ~ 1757668936 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricSnapshots={clock_diff_milliseconds=[SeriesMetricValues(metric=Metric{name='clock_diff_milliseconds', labels={app=HOST, host=192.168.2.181, host_name=localhost.localdomain, ob_cluster_id=0, ob_cluster_name=, svr_ip=192.168.2.181}}, data=[ValueData(timestamp=1757668920, value=2104612.0)])]}, metricEvaluateResults={clock_diff_milliseconds > 50=[ExpressionEvaluateResult(matchValue=50.0, metric=clock_diff_milliseconds, match=true, value=2104612.0, values={1757668920=2104612.0}, valueCount=1, labels={app=HOST, host=192.168.2.181, host_name=localhost.localdomain, ob_cluster_id=0, ob_cluster_name=, svr_ip=192.168.2.181})]}, matched=true, isContainValidValue=null, message=null, level=1) 2025-09-12 17:22:26.659 INFO 6621 --- [alarm-detect-18,19befb8ddd3b8207,49ce91372098dd25] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ocp_remote_server_time_diff_too_large, target=alarm_template_id=0:host=192.168.2.171-1, currentState=Firing 2025-09-12 17:22:26.659 INFO 6621 --- [alarm-detect-18,19befb8ddd3b8207,49ce91372098dd25] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ocp_remote_server_time_diff_too_large, target=alarm_template_id=0:host=192.168.2.181-1, currentState=Firing 2025-09-12 17:22:26.660 INFO 6621 --- [alarm-detect-18,19befb8ddd3b8207,49ce91372098dd25] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ocp_remote_server_time_diff_too_large, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:host=192.168.2.161-1, alarm_template_id=0:host=192.168.2.171-1, alarm_template_id=0:host=192.168.2.181-1] 2025-09-12 17:22:26.681 INFO 6621 --- [alarm-detect-18,19befb8ddd3b8207,49ce91372098dd25] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=4 2025-09-12 17:22:26.705 INFO 6621 --- [alarm-detect-18,19befb8ddd3b8207,49ce91372098dd25] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=5 2025-09-12 17:22:26.829 INFO 6621 --- [alarm-task-2,33c89928ce55b7c6,a5469c33d3b1e9ff] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-09-12T17:22:26.828402+08:00, lastProcessTime=2025-09-12T17:22:16.825088+08:00 2025-09-12 17:22:26.992 INFO 6621 --- [alarm-task-2,33c89928ce55b7c6,a5469c33d3b1e9ff] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-09-12T17:22:26.828402+08:00 2025-09-12 17:22:27.433 INFO 6621 --- [ocp-schedules-4,5000bd77b60d56cf,028157ee68109bdf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-09-12 17:22:28.244 INFO 6621 --- [prometheus-minute-schedule-1,a792d777d21e97c6,915f44a5a3c58a6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1757668980, interval=60 2025-09-12 17:22:28.248 INFO 6621 --- [prometheus-minute-schedule-1,a792d777d21e97c6,915f44a5a3c58a6a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2025-09-12 17:22:28.396 INFO 6621 --- [ocp-async-1,e3a3251a49941a38,419081555c2ad5bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:28.620 WARN 6621 --- [ocp-async-9,bec7c318c6e5f0c1,3b17b7dd40864ced] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:28.844 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.2.185:8080 refreshes policies... 2025-09-12 17:22:28.845 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-09-12 17:22:29.137 INFO 6621 --- [ocp-async-1,34954871930eb5c8,b9a53117779d75b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-09-12 17:22:30.397 INFO 6621 --- [ocp-async-10,156d1bea1dbdf683,e53cc5585f401a5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:30.620 WARN 6621 --- [ocp-async-6,ea92d9f4c57a1ee5,43b8b42b0f47035f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:30.714 INFO 6621 --- [prometheus-second-schedule-1,c842e0485ec9d8c8,73fe481ddee3af2a] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=3, collectTaskCount=3 2025-09-12 17:22:31.244 INFO 6621 --- [prometheus-minute-schedule-1,809b1cac0497836f,99c2aef162a8d730] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1757668980, interval=60 2025-09-12 17:22:31.250 INFO 6621 --- [prometheus-minute-schedule-1,809b1cac0497836f,99c2aef162a8d730] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2025-09-12 17:22:31.846 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.2.185:8080 refreshes policies... 2025-09-12 17:22:31.847 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-09-12 17:22:32.397 INFO 6621 --- [ocp-async-6,b6a7826370094ade,beb47deb6db4f8ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:32.620 WARN 6621 --- [ocp-async-1,5d6fc857eb59de38,a2748e61c565ad1e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:34.244 INFO 6621 --- [prometheus-minute-schedule-1,0957bebc40fa7e59,a7e35284d01bf754] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1757668980, interval=60 2025-09-12 17:22:34.246 INFO 6621 --- [prometheus-minute-schedule-1,0957bebc40fa7e59,a7e35284d01bf754] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2025-09-12 17:22:34.397 INFO 6621 --- [ocp-async-1,5f0b2d57d4f4acf0,5e5bc9457035c955] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:34.474 INFO 6621 --- [ocp-schedules-3,fb41566177684664,b6495070ffc2f657] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-09-12 17:22:34.477 INFO 6621 --- [ocp-schedules-3,fb41566177684664,b6495070ffc2f657] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-09-12 17:22:34.620 WARN 6621 --- [ocp-async-9,e36ab7a69ce1f093,43edeaa6e228068d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:34.847 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.2.185:8080 refreshes policies... 2025-09-12 17:22:34.847 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-09-12 17:22:36.397 INFO 6621 --- [ocp-async-7,364572d8931eedb7,8d2ec5fe1a6c114c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:36.620 WARN 6621 --- [ocp-async-2,e295b34e33c3e58d,db7c203edad842ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:36.714 INFO 6621 --- [ocp-schedules-9,6799cdc4bc3f7149,ff39869e39ee1292] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-09-12 17:22:36.734 WARN 6621 --- [scanner-1340351-1,6799cdc4bc3f7149,cf7335a8e78b0e5e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 6 2025-09-12 17:22:36.832 INFO 6621 --- [alarm-task-2,33c89928ce55b7c6,a5469c33d3b1e9ff] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-09-12T17:22:36.831649+08:00, lastProcessTime=2025-09-12T17:22:26.828402+08:00 2025-09-12 17:22:36.849 INFO 6621 --- [ocp-schedules-15,735bdf9bb41a1617,118e93a035923eb6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-09-12 17:22:36.856 INFO 6621 --- [ocp-schedules-15,735bdf9bb41a1617,118e93a035923eb6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[bd9d56f1-2d09-45eb-984f-2776b048c82b] 2025-09-12 17:22:36.856 INFO 6621 --- [ocp-schedules-15,735bdf9bb41a1617,118e93a035923eb6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[bd9d56f1-2d09-45eb-984f-2776b048c82b] 2025-09-12 17:22:36.984 INFO 6621 --- [alarm-task-2,33c89928ce55b7c6,a5469c33d3b1e9ff] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-09-12T17:22:36.831649+08:00 2025-09-12 17:22:37.244 INFO 6621 --- [prometheus-minute-schedule-1,331a8b162696e0cb,d81a4ec6f2fdcb12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1757668980, interval=60 2025-09-12 17:22:37.246 INFO 6621 --- [prometheus-minute-schedule-1,331a8b162696e0cb,d81a4ec6f2fdcb12] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2025-09-12 17:22:37.438 INFO 6621 --- [ocp-schedules-25,fd0227925535c1ca,07f5e7c846a43c21] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-09-12 17:22:37.847 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.2.185:8080 refreshes policies... 2025-09-12 17:22:37.848 INFO 6621 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-09-12 17:22:38.397 INFO 6621 --- [ocp-async-3,1dba66ff72ef625c,a0d3465bbaddc9e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-09-12 17:22:38.548 INFO 6621 --- [ocp-schedules-22,eeb7c91cd1d3ee20,40ace2b49851a753] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-09-12 17:22:38.550 INFO 6621 --- [ocp-schedules-16,43a6e669808aed66,88c69525fa4f7395] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[192.168.2.171, 192.168.2.161, 192.168.2.181] 2025-09-12 17:22:38.551 INFO 6621 --- [host-check-agent-status1,43a6e669808aed66,394598a48d5eb7a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=192.168.2.171, originalStatus=AVAILABLE 2025-09-12 17:22:38.551 INFO 6621 --- [host-check-agent-status2,43a6e669808aed66,0f7e4fca1606501f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=192.168.2.161, originalStatus=AVAILABLE 2025-09-12 17:22:38.551 INFO 6621 --- [host-check-agent-status3,43a6e669808aed66,f251d7e8797631ee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=192.168.2.181, originalStatus=AVAILABLE 2025-09-12 17:22:38.551 INFO 6621 --- [ocp-schedules-7,4c8be4e73ec14f52,7cabc0521e024b3f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[192.168.2.171, 192.168.2.161, 192.168.2.181] 2025-09-12 17:22:38.551 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=192.168.2.171, originalStatus=AVAILABLE 2025-09-12 17:22:38.551 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=192.168.2.161, originalStatus=AVAILABLE 2025-09-12 17:22:38.551 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=192.168.2.181, originalStatus=AVAILABLE 2025-09-12 17:22:38.553 INFO 6621 --- [ocp-schedules-11,de52553b90c3c830,af388b671989d99a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[192.168.2.171, 192.168.2.161, 192.168.2.181] 2025-09-12 17:22:38.554 INFO 6621 --- [host-check-clockdiff1,de52553b90c3c830,90debe6a8e552d4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.2.171], timeoutMillis=20000 2025-09-12 17:22:38.554 INFO 6621 --- [host-check-clockdiff2,de52553b90c3c830,cb124293fb8ad91e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.2.161], timeoutMillis=20000 2025-09-12 17:22:38.554 INFO 6621 --- [host-check-clockdiff3,de52553b90c3c830,d1b7d1c0e37bc2b4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.2.181], timeoutMillis=20000 2025-09-12 17:22:38.560 INFO 6621 --- [host-check-agent-status1,43a6e669808aed66,394598a48d5eb7a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-09-12 17:22:38.560 INFO 6621 --- [ocp-schedules-30,bb688655c7fe579e,d401b5e9e5f88dee] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-09-12 17:22:38.560 INFO 6621 --- [ocp-schedules-12,96d4f1519a7d5f0a,139c02df8e7559d6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-09-12 17:22:38.560 INFO 6621 --- [ocp-schedules-24,4a4c1699609a5eeb,9f236e2aedf903af] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-09-12 17:22:38.563 INFO 6621 --- [ocp-schedules-12,96d4f1519a7d5f0a,139c02df8e7559d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-09-12 17:22:38.563 INFO 6621 --- [host-check-agent-status1,43a6e669808aed66,394598a48d5eb7a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-09-12 17:22:38.563 INFO 6621 --- [ocp-schedules-30,bb688655c7fe579e,d401b5e9e5f88dee] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-09-12 17:22:38.563 INFO 6621 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ztocp, obClusterId=1757648497, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-09-12T11:52:54+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/data/1","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/data/log1","SqlPort":2881,"SvrPort":2882}, tenantCount=0) 2025-09-12 17:22:38.564 INFO 6621 --- [host-check-clockdiff1,de52553b90c3c830,90debe6a8e552d4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1757668958 -2104612 -2104612 , error= 2025-09-12 17:22:38.565 INFO 6621 --- [host-check-agent-status2,43a6e669808aed66,0f7e4fca1606501f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-09-12 17:22:38.565 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-09-12 17:22:38.565 INFO 6621 --- [ocp-schedules-24,4a4c1699609a5eeb,9f236e2aedf903af] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-09-12 17:22:38.565 INFO 6621 --- [ocp-schedules-24,4a4c1699609a5eeb,9f236e2aedf903af] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-09-12 17:22:38.566 INFO 6621 --- [ocp-schedules-30,bb688655c7fe579e,d401b5e9e5f88dee] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-09-12 17:22:38.566 INFO 6621 --- [host-check-agent-status3,43a6e669808aed66,f251d7e8797631ee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-09-12 17:22:38.566 INFO 6621 --- [host-check-agent-status2,43a6e669808aed66,0f7e4fca1606501f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-09-12 17:22:38.567 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-09-12 17:22:38.568 INFO 6621 --- [host-check-agent-status3,43a6e669808aed66,f251d7e8797631ee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-09-12 17:22:38.568 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-09-12 17:22:38.569 INFO 6621 --- [ocp-schedules-18,a9e0e34e8f1fdf20,003c8581bb8ff7cb] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-09-12 17:22:38.569 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-09-12 17:22:38.570 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-09-12 17:22:38.571 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-09-12 17:22:38.577 INFO 6621 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-09-12 17:22:38.577 INFO 6621 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ztocp, obClusterId=1757648497, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-09-12T11:52:54+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/data/1","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/data/log1","SqlPort":2881,"SvrPort":2882}, tenantCount=0) 2025-09-12 17:22:38.578 INFO 6621 --- [ocp-schedules-12,96d4f1519a7d5f0a,139c02df8e7559d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-09-12 17:22:38.587 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.181:62888/api/v1/time, params:null 2025-09-12 17:22:38.588 INFO 6621 --- [host-check-agent-status1,43a6e669808aed66,394598a48d5eb7a2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.171:62888/api/v1/agent/status, params:null 2025-09-12 17:22:38.589 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=192.168.2.181, timestamp=2025-09-12T16:47:33.977688505+08:00 2025-09-12 17:22:38.590 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-09-12T17:22:38.589+08:00 2025-09-12 17:22:38.591 INFO 6621 --- [host-check-agent-status2,43a6e669808aed66,0f7e4fca1606501f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.161:62888/api/v1/agent/status, params:null 2025-09-12 17:22:38.592 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.171:62888/api/v1/time, params:null 2025-09-12 17:22:38.592 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-09-12T17:22:38.589+08:00 and status AVAILABLE 2025-09-12 17:22:38.593 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=192.168.2.171, timestamp=2025-09-12T16:47:33.981068943+08:00 2025-09-12 17:22:38.593 INFO 6621 --- [host-check-agent-status3,43a6e669808aed66,f251d7e8797631ee] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.181:62888/api/v1/agent/status, params:null 2025-09-12 17:22:38.593 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-09-12T17:22:38.593+08:00 2025-09-12 17:22:38.594 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.2.161:62888/api/v1/time, params:null 2025-09-12 17:22:38.594 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-09-12 17:22:38.594 INFO 6621 --- [host-check-agent-status1,43a6e669808aed66,394598a48d5eb7a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=192.168.2.171 2025-09-12 17:22:38.594 INFO 6621 --- [host-check-agent-status1,43a6e669808aed66,394598a48d5eb7a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=192.168.2.171 2025-09-12 17:22:38.594 INFO 6621 --- [host-check-agent-status1,43a6e669808aed66,394598a48d5eb7a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=192.168.2.171 2025-09-12 17:22:38.595 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=192.168.2.161, timestamp=2025-09-12T16:47:33.983620092+08:00 2025-09-12 17:22:38.595 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-09-12T17:22:38.593+08:00 and status AVAILABLE 2025-09-12 17:22:38.596 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-09-12T17:22:38.595+08:00 2025-09-12 17:22:38.596 INFO 6621 --- [host-check-agent-status3,43a6e669808aed66,f251d7e8797631ee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=192.168.2.181 2025-09-12 17:22:38.596 INFO 6621 --- [host-check-agent-status3,43a6e669808aed66,f251d7e8797631ee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=192.168.2.181 2025-09-12 17:22:38.596 INFO 6621 --- [host-check-agent-status3,43a6e669808aed66,f251d7e8797631ee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=192.168.2.181 2025-09-12 17:22:38.597 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-09-12 17:22:38.597 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-09-12T17:22:38.595+08:00 and status AVAILABLE 2025-09-12 17:22:38.599 INFO 6621 --- [host-check-agent-status2,43a6e669808aed66,0f7e4fca1606501f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=192.168.2.161 2025-09-12 17:22:38.599 INFO 6621 --- [host-check-agent-status2,43a6e669808aed66,0f7e4fca1606501f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=192.168.2.161 2025-09-12 17:22:38.599 INFO 6621 --- [host-check-agent-status2,43a6e669808aed66,0f7e4fca1606501f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=192.168.2.161 2025-09-12 17:22:38.599 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-09-12 17:22:38.600 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-09-12 17:22:38.601 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-09-12 17:22:38.604 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-09-12 17:22:38.605 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-09-12 17:22:38.606 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-09-12 17:22:38.607 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-09-12 17:22:38.616 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.181:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-09-12 17:22:38.621 WARN 6621 --- [ocp-async-5,af6a7b9ecede01a7,70c0050082a86f3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-09-12 17:22:38.623 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.171:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-09-12 17:22:38.623 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.161:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-09-12 17:22:38.651 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-09-12 17:22:38.660 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-09-12 17:22:38.663 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-09-12 17:22:38.670 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-09-12 17:22:38.672 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.181:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-09-12 17:22:38.675 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-09-12 17:22:38.682 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-09-12 17:22:38.689 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.171:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-09-12 17:22:38.697 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.161:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-09-12 17:22:38.710 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-09-12 17:22:38.717 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-09-12 17:22:38.728 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.181:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-09-12 17:22:38.731 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-09-12 17:22:38.740 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-09-12 17:22:38.747 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-09-12 17:22:38.758 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.171:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-09-12 17:22:38.759 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-09-12 17:22:38.769 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.2.161:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-09-12 17:22:38.772 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-09-12 17:22:38.779 INFO 6621 --- [host-check-host-status3,4c8be4e73ec14f52,690a633a70d423e3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-09-12 17:22:38.804 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-09-12 17:22:38.811 INFO 6621 --- [host-check-host-status1,4c8be4e73ec14f52,09845e9efa99e059] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-09-12 17:22:38.813 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-09-12 17:22:38.822 INFO 6621 --- [host-check-host-status2,4c8be4e73ec14f52,3870f3c738f48ca8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-09-12 17:22:39.137 INFO 6621 --- [ocp-async-3,5805324e65556323,1b7e3a8f05aaeee9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-09-12 17:22:39.498 INFO 6621 --- [host-check-clockdiff3,de52553b90c3c830,d1b7d1c0e37bc2b4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1757668959 -2104612 -2104612 , error= 2025-09-12 17:22:39.500 INFO 6621 --- [host-check-clockdiff2,de52553b90c3c830,cb124293fb8ad91e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1757668959 -2104612 -2104612 , error=