2024-05-24 19:29:44.319 INFO 5670 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.0.111:8080 refreshes policies... 2024-05-24 19:29:44.319 INFO 5670 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-05-24 19:29:44.378 WARN 5670 --- [SimpleAsyncTaskExecutor-32279,,] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-05-24 19:29:44.421 INFO 5670 --- [pool-Check obproxy1,f60df38b2bfa4345,62b20503d7b5] c.o.o.s.o.o.SystemOdpOperatorFactory : connect obproxy 192.168.0.72 2883 2024-05-24 19:29:44.422 INFO 5670 --- [pool-Check obproxy3,f60df38b2bfa4345,042c7ec40cee] c.o.o.s.o.o.SystemOdpOperatorFactory : connect obproxy 192.168.0.66 2883 2024-05-24 19:29:44.422 INFO 5670 --- [pool-Check obproxy2,f60df38b2bfa4345,e7382af787d0] c.o.o.s.o.o.SystemOdpOperatorFactory : connect obproxy 192.168.0.71 2883 2024-05-24 19:29:44.471 INFO 5670 --- [SimpleAsyncTaskExecutor-32282,,] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-05-24 19:29:45.101 INFO 5670 --- [pool-prometheus-second-schedule-1,d7597f8834874da4,9afccdcc3245] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2024-05-24 19:29:46.374 WARN 5670 --- [SimpleAsyncTaskExecutor-32302,,] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-05-24 19:29:46.471 INFO 5670 --- [SimpleAsyncTaskExecutor-32305,,] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-05-24 19:29:47.320 INFO 5670 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.0.111:8080 refreshes policies... 2024-05-24 19:29:47.321 INFO 5670 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-05-24 19:29:48.375 WARN 5670 --- [SimpleAsyncTaskExecutor-32325,,] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-05-24 19:29:48.472 INFO 5670 --- [SimpleAsyncTaskExecutor-32328,,] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-05-24 19:29:49.092 INFO 5670 --- [SimpleAsyncTaskExecutor-32335,,] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-05-24 19:29:49.122 WARN 5670 --- [pool-scanner-1117723-1,72efd3c1a76746c0,037bd627a4ec] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 21 2024-05-24 19:29:49.505 INFO 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/backup/config/storages/test, client 192.168.0.13, traceId 6bf542226c07481b] 2024-05-24 19:29:49.506 INFO 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/backup/config/storages/test?null, client=192.168.0.13, traceId=6bf542226c07481b, method=NoDataResponse com.oceanbase.ocp.server.common.controller.ObBackupController.testStorageConfig(TestStorageParam), args=TestStorageParam(hostIdList=null, clusterId=1, obBackupStorageBaseInfo=ObBackupStorageBaseInfo(backupStorageType=BACKUP_STORAGE_FILE, storageUrl=/obbackup/, ossAccessKey=null, cosAccessKey=null), backupPurposeType=BACKUP, storageId=1000004),] 2024-05-24 19:29:49.506 WARN 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.b.i.u.BackupParamSecurityUtils : storage[BACKUP_STORAGE_FILE] not need decrypt param 2024-05-24 19:29:49.506 INFO 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.b.i.f.c.BackupCommonServiceImpl : test storage config, type=file, url=/obbackup/ 2024-05-24 19:29:49.538 INFO 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-05-24 19:29:49.539 INFO 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-05-24 19:29:49.552 INFO 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.0.66:62888/api/v1/file/checkDirectoryPermission, request body:CheckDirectoryPermissionRequest(directory=/obbackup/, user=admin, permission=ACCESS_WRITE, onlyExistingAncestor=false), params:null 2024-05-24 19:29:49.560 INFO 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-05-24 19:29:49.561 INFO 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-05-24 19:29:49.571 INFO 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.0.71:62888/api/v1/file/checkDirectoryPermission, request body:CheckDirectoryPermissionRequest(directory=/obbackup/, user=admin, permission=ACCESS_WRITE, onlyExistingAncestor=false), params:null 2024-05-24 19:29:49.578 INFO 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-05-24 19:29:49.579 INFO 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-05-24 19:29:49.587 INFO 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://192.168.0.72:62888/api/v1/file/checkDirectoryPermission, request body:CheckDirectoryPermissionRequest(directory=/obbackup/, user=admin, permission=ACCESS_WRITE, onlyExistingAncestor=false), params:null 2024-05-24 19:29:49.597 ERROR 5670 --- [http-nio-0.0.0.0-8080-exec-6,6bf542226c07481b,cb18ed2ab871] c.o.o.s.c.trace.RequestTracingAspect : Unexpected API Error: [POST /api/v2/backup/config/storages/test client=192.168.0.13, traceId=6bf542226c07481b, args=TestStorageParam(hostIdList=null, clusterId=1, obBackupStorageBaseInfo=ObBackupStorageBaseInfo(backupStorageType=BACKUP_STORAGE_FILE, storageUrl=/obbackup/, ossAccessKey=null, cosAccessKey=null), backupPurposeType=BACKUP, storageId=1000004),, duration=94 ms] com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=BACKUP_CONFIG_STORAGE_TEST_FAIL, args=hostIpList:192.168.0.66,192.168.0.71,192.168.0.72,file:///obbackup/ at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl.checkFailedHostList(BackupCommonServiceImpl.java:633) at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl.testStorageConfig(BackupCommonServiceImpl.java:545) at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl$$FastClassBySpringCGLIB$$c14a6096.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:704) at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl$$EnhancerBySpringCGLIB$$53001362.testStorageConfig() at com.oceanbase.ocp.server.common.controller.ObBackupController.testStorageConfig(ObBackupController.java:175) at com.oceanbase.ocp.server.common.controller.ObBackupController$$FastClassBySpringCGLIB$$5f20ca8d.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.oceanbase.ocp.server.common.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:130) at sun.reflect.GeneratedMethodAccessor839.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at com.oceanbase.ocp.server.common.controller.ObBackupController$$EnhancerBySpringCGLIB$$416681b1.testStorageConfig() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:528) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:596) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:67) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:106) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:132) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) at com.oceanbase.ocp.server.common.trace.TraceFilter.doFilter(TraceFilter.java:47) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:750) 2024-05-24 19:29:49.657 INFO 5670 --- [alarm-task-1,1b867af99d794eec,a3a859b90632] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-05-24T19:29:49.656576+08:00, lastProcessTime=2024-05-24T19:29:39.701763+08:00 2024-05-24 19:29:49.717 INFO 5670 --- [alarm-task-1,1b867af99d794eec,a3a859b90632] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-05-24T19:29:49.656576+08:00 2024-05-24 19:29:49.834 INFO 5670 --- [SimpleAsyncTaskExecutor-32344,,] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-05-24 19:29:50.321 INFO 5670 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.0.111:8080 refreshes policies... 2024-05-24 19:29:50.321 INFO 5670 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-05-24 19:29:50.375 WARN 5670 --- [SimpleAsyncTaskExecutor-32351,,] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-05-24 19:29:50.472 INFO 5670 --- [SimpleAsyncTaskExecutor-32354,,] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-05-24 19:29:50.904 INFO 5670 --- [pool-ocp-schedules-9,7d89c5a52ecc45ab,1a6c3bd6f40a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters